Summary: Differential equation (DE) is a commonly used modeling method in various scientific subjects such as finance and biology. The parameters in DE models often have interesting scientific interpretations, but their values are often unknown and need to be estimated from the measurements. Here, we develop a one-stage parameter estimation framework, which is based on the Markov Chain Monte Carlo (MCMC) method, to draw the samples from the posterior distribution of the unknown parameters from the given noisy and scarce observations of the solution only. A likelihood function including a novel potential is constructed to infer the unknown parameters and the novel potential works by measuring the residual errors of both data and DE model with given model parameters. A key issue in parameter estimation problem is to robustly estimate the solution and its derivatives from noisy observations of only the function values at given location points, under the assumption of a physical model in the form of differential equation governing the function and its derivatives. To address the key issue, we propose to use the Gaussian process regression with constraint (GPRC) method which jointly model the solution, its derivatives, and the parametric differential equation, to estimate the solution and its derivatives. With numerical examples, we illustrate that the proposed method has competitive performance against existing approaches for estimating the unknown parameters in DEs.