×

Finite-dimensional approximation of Gaussian processes with linear inequality constraints and noisy observations. (English) Zbl 07753686

Summary: Due to their flexibility, Gaussian processes (GPs) have been widely used in nonparametric function estimation. A prior information about the underlying function is often available. In this article, the finite-dimensional Gaussian approach proposed by Maatouk and Bay in 2017 which can satisfy linear inequality conditions everywhere (e.g., monotonicity, convexity and boundary) is considered. In a variety of real-world problems, the observed data usually possess noise. In this article, this approach has been extended to deal with noisy observations. The mean and the maximum of the posterior distribution are well defined. Additionally, to simulate from the posterior distribution two methods have been used: the exact rejection sampling from the Mode and the Hamiltonian Monte Carlo method which is more efficient in high-dimensional cases. The generalization of the Kimeldorf-Wahba correspondence is proved in noisy observation cases. A comparison shown that the proposed model outperforms all recent models dealing with the same constraints in terms of predictive accuracy and coverage intervals.

MSC:

62-XX Statistics

References:

[1] Abrahamsen, P.; Benth, F. E., Kriging with inequality constraints, Mathematical Geology, 33, 6, 719-44 (2001) · Zbl 1011.86007 · doi:10.1023/A:1011078716252
[2] Agrell, C., Gaussian processes with linear operator inequality constraints, Journal of Machine Learning Research, 20, 1-36 (2019) · Zbl 1441.62179
[3] Aronszajn, N., Theory of reproducing kernels, Transactions of the American Mathematical Society, 68, 3, 337-404 (1950) · Zbl 0037.20701 · doi:10.1090/S0002-9947-1950-0051437-7
[4] Bachoc, F.; Lagnoux, A.; López-Lopera, A. F., Maximum likelihood estimation for Gaussian processes under inequality constraints, Electronic Journal of Statistics, 13, 2, 2921-69 (2019) · Zbl 1428.62420 · doi:10.1214/19-EJS1587
[5] Bay, X.; Grammont, L.; Maatouk, H., Generalization of the Kimeldorf-Wahba correspondence for constrained interpolation, Electronic Journal of Statistics, 10, 1, 1580-95 (2016) · Zbl 1348.60055 · doi:10.1214/16-EJS1149
[6] Bay, X.; Grammont, L.; Maatouk, H., A new method for interpolating in a convex subset of a Hilbert space, Computational Optimization and Applications, 68, 1, 95-120 (2017) · Zbl 1406.90127 · doi:10.1007/s10589-017-9906-9
[7] Berlinet, A.; Thomas-Agnan, C., Reproducing kernel Hilbert spaces in probability and statistics (2004), Boston, MA: Kluwer Academic Publishers, Boston, MA · Zbl 1145.62002
[8] Bhattacharya, R.; Kong, M., Consistency and asymptotic normality of the estimated effective doses in bioassay, Journal of Statistical Planning and Inference, 137, 3, 643-58 (2007) · Zbl 1104.62117 · doi:10.1016/j.jspi.2006.06.027
[9] Bhattacharya, R.; Lin, L., An adaptive nonparametric method in benchmark analysis for bioassay and environmental studies, Statistics & Probability Letters, 80, 23-24, 1947-53 (2010) · Zbl 1202.62145
[10] Bhattacharya, R.; Lin, L., Nonparametric benchmark analysis in risk assessment: A comparative study by simulation and data analysis, Sankhya. Series B [Methodological], 73, 1, 144-63 (2011) · Zbl 1271.62062 · doi:10.1007/s13571-011-0019-7
[11] Bornkamp, B.; Ickstadt, K., Bayesian nonparametric estimation of continuous monotone functions with applications to dose-response analysis, Biometrics, 65, 1, 198-205 (2009) · Zbl 1159.62023 · doi:10.1111/j.1541-0420.2008.01060.x
[12] Botts, C., An accept-reject algorithm for the positive multivariate normal distribution, Computational Statistics, 28, 4, 1749-73 (2013) · Zbl 1306.65032 · doi:10.1007/s00180-012-0377-2
[13] Boyd, S.; Vandenberghe, L., Convex optimization (2004), Cambridge: Cambridge University Press, Cambridge · Zbl 1058.90049
[14] Barlow, H. R.; Bartholomew, D.; Bremner, J., Statistical inference under order restrictions (The theory and application of isotonic regression) (1972), Missouri Univ Columbia Dept of Statistics · Zbl 0246.62038
[15] Chopin, N., Fast simulation of truncated Gaussian distributions, Statistics and Computing, 21, 2, 275-88 (2011) · Zbl 1284.65015 · doi:10.1007/s11222-009-9168-1
[16] Cousin, A.; Maatouk, H.; Rullière, D., Kriging of financial term-structures, European Journal of Operational Research, 255, 2, 631-48 (2016) · Zbl 1346.91264 · doi:10.1016/j.ejor.2016.05.057
[17] Cramer, H.; Leadbetter, R., Stationary and related stochastic processes: Sample function properties and their applications (1967), New York: Wiley, New York · Zbl 0162.21102
[18] Da Veiga, S.; Marrel, A., Gaussian process modeling with inequality constraints, Annales de la Faculté Des Sciences de Toulouse: Mathématiques, 21, 3, 529-55 (2012) · Zbl 1279.60047 · doi:10.5802/afst.1344
[19] Delecroix, M.; Simioni, M.; Thomas-Agnan, C., Functional estimation under shape constraints, Journal of Nonparametric Statistics, 6, 1, 69-89 (1996) · Zbl 0862.62034 · doi:10.1080/10485259608832664
[20] Dette, H.; Neumeyer, N.; Pilz, K. F., A note on nonparametric estimation of the effective dose in quantal bioassay, Journal of the American Statistical Association, 100, 470, 503-10 (2005) · Zbl 1117.62317 · doi:10.1198/016214504000001493
[21] Freulon, X.; de Fouquet, C.; Soares, A., Geostatistics Tróia ’92, 1, Conditioning a Gaussian model with inequalities, 201-12 (1993), Dordrecht: Springer Netherlands, Dordrecht
[22] Gerhard, T.; Florian, L., Generalized smooth monotonic regression in additive modeling, Journal of Computational and Graphical Statistics, 16, 1, 165-88 (2007)
[23] Golchi, S.; Bingham, D.; Chipman, H.; Campbell, D., Monotone emulation of computer experiments, SIAM/ASA Journal on Uncertainty Quantification, 3, 1, 370-92 (2015) · Zbl 1327.62146 · doi:10.1137/140976741
[24] Goldfarb, D.; Idnani, A., A numerically stable dual method for solving strictly convex quadratic programs, Mathematical Programming, 27, 1, 1-33 (1983) · Zbl 0537.90081 · doi:10.1007/BF02591962
[25] Maatouk, L. H.; Bay, X. (2022)
[26] Holmes, C.; Heard, N., Generalized monotonic regression using random change points, Statistics in Medicine, 22, 4, 623-38 (2003)
[27] Nielsen, B. S. J. B.; Larsen, J., Bounded Gaussian process regression, 2013 IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 1-6 (2013), Southhampton, UK: IEEE, Southhampton, UK · doi:10.1109/MLSP.2013.6661916
[28] Jidling, C.; Wahlström, N.; Wills, A.; Schön, T. B., Linearly constrained Gaussian processes, 1215-24 (2017)
[29] Kimeldorf, G. S.; Wahba, G., A correspondence between Bayesian estimation on stochastic processes and smoothing by splines, The Annals of Mathematical Statistics, 41, 2, 495-502 (1970) · Zbl 0193.45201 · doi:10.1214/aoms/1177697089
[30] Kleijnen, J. P.; Van Beers, W. C., Monotonicity-preserving bootstrapped kriging metamodels for expensive simulations, Journal of the Operational Research Society, 64, 5, 708-17 (2013) · doi:10.1057/jors.2011.148
[31] Kong, M.; Eubank, R. L., Monotone smoothing with application to dose-response curve, Communications in Statistics - Simulation and Computation, 35, 4, 991-1004 (2006) · Zbl 1105.62039 · doi:10.1080/03610910600880492
[32] Lee, O. C.; Ghosh, J., Constrained Gaussian process regression for gene-disease association, 72-9 (2013), Dallas, TX: IEEE, Dallas, TX
[33] Lin, L.; Dunson, D. B., Bayesian monotone regression using Gaussian process projection, Biometrika, 101, 2, 303-17 (2014) · Zbl 1452.62285 · doi:10.1093/biomet/ast063
[34] López-Lopera, A.; Bachoc, F.; Durrande, N.; Roustant, O., Finite-dimensional Gaussian approximation with linear inequality constraints, SIAM/ASA Journal on Uncertainty Quantification, 6, 3, 1224-55 (2018) · Zbl 1405.60047 · doi:10.1137/17M1153157
[35] Lyu, X.; Binois, M.; Ludkovski, M., Evaluating Gaussian process metamodels and sequential designs for noisy level set estimation, Statistics and Computing, 31, 4, 1-21 (2021) · Zbl 1475.62048
[36] Maatouk, H.; Bay, X.; Cools, R.; Nuyens, D., Monte Carlo and quasi-Monte Carlo methods, A new rejection sampling method for truncated multivariate Gaussian random variables restricted to convex sets, 521-30 (2016), Cham: Springer International Publishing, Cham · Zbl 1356.65012
[37] Maatouk, H.; Bay, X., Gaussian process emulators for computer experiments with inequality constraints, Mathematical Geosciences, 49, 5, 557-82 (2017) · Zbl 1371.65006 · doi:10.1007/s11004-017-9673-2
[38] Maatouk, H.; Roustant, O.; Richet, Y., Cross-validation estimations of hyper-parameters of Gaussian processes with inequality constraints, Procedia Environmental Sciences, 27, 38-44 (2015) · doi:10.1016/j.proenv.2015.07.105
[39] Mammen, E., Estimating a smooth monotone regression function, Annals of Statistics, 19, 2, 724-40 (1991) · Zbl 0737.62038
[40] Morris, M. D.; Mitchell, T. J.; Ylvisaker, D., Bayesian design and analysis of computer experiments: Use of derivatives in surface prediction, Technometrics, 35, 3, 243-55 (1993) · Zbl 0785.62025 · doi:10.1080/00401706.1993.10485320
[41] Müller, H.-G; Schmitt, T., Kernel and probit estimates in quantal bioassay, Journal of the American Statistical Association, 83, 403, 750-9 (1988) · Zbl 0662.62112
[42] Murphy, K. P., Machine learning: A probabilistic perspective (adaptive computation and machine learning series) (2012), Cambridge: The MIT Press, Cambridge · Zbl 1295.68003
[43] Neelon, B.; Dunson, D. B., Bayesian isotonic regression and trend analysis, Biometrics, 60, 2, 398-406 (2004) · Zbl 1125.62023 · doi:10.1111/j.0006-341X.2004.00184.x
[44] Pakman, A.; Paninski, L., Exact Hamiltonian Monte Carlo for truncated multivariate Gaussians, Journal of Computational and Graphical Statistics, 23, 2, 518-42 (2014) · doi:10.1080/10618600.2013.788448
[45] Parzen, E., Stochastic processes. Holden-Day series in probability and statistics (1962), San Francisco, London, Amsterdam: Holden-Day, San Francisco, London, Amsterdam · Zbl 0107.12301
[46] Philippe, A.; Robert, C. P., Perfect simulation of positive Gaussian distributions, Statistics and Computing, 13, 2, 179-86 (2003) · doi:10.1023/A:1023264710933
[47] Ramsay, J. O., Monotone regression splines in action, Statistical Science, 3, 4, 425-41 (1988)
[48] Ramsay, J. O., Estimating smooth monotone functions, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 60, 2, 365-75 (1998) · Zbl 0909.62041 · doi:10.1111/1467-9868.00130
[49] Rasmussen, C. E.; Williams, C. K., Gaussian processes for machine learning (2006), Cambridge: MIT Press, Cambridge · Zbl 1177.68165
[50] Riihimäki, J.; Vehtari, A., Gaussian processes with monotonicity information, Journal of Machine Learning Research, 9, 645-52 (2010)
[51] Robert, C. P., Simulation of truncated normal variables, Statistics and Computing, 5, 2, 121-5 (1995) · doi:10.1007/BF00143942
[52] Wright, T. F. T.; Dykstra, R. L., Order restricted statistical inference (1988), New York: Wiley, New York · Zbl 0645.62028
[53] Rudovic, O.; Pantic, M., Shape-constrained Gaussian process regression for facial-point-based head-pose normalization, In 2011 IEEE International Conference on Computer Vision (ICCV). -, 1495-1502 (2011), Barcelona, Spain: IEEE, Barcelona, Spain
[54] Saarela, O.; Arjas, E., A method for Bayesian monotonic multiple regression, Scandinavian Journal of Statistics, 38, 3, 499-513 (2011) · Zbl 1246.62111
[55] Salzmann, M.; Urtasun, R.; Lafferty, J. D.; Williams, C. K. I.; Shawe-Taylor, J.; Zemel, R. S.; Culotta, A., Advances in neural information processing systems, 23, Implicitly constrained Gaussian process regression for monocular non-rigid pose estimation, 2065-73 (2010), New York: Curran Associates, Inc, New York
[56] Shively, T. S.; Sager, T. W.; Walker, S. G., A Bayesian approach to non-parametric monotone function estimation, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 71, 1, 159-75 (2009) · Zbl 1231.62058 · doi:10.1111/j.1467-9868.2008.00677.x
[57] Bhattacharya, A. A.; Pati, D. (2018)
[58] Swiler, L. P.; Gulian, M.; Frankel, A. L.; Safta, C.; Jakeman, J. D., A survey of constrained Gaussian process regression: Approaches and implementation challenges, Journal of Machine Learning for Modeling and Computing, 1, 2, 119-56 (2020) · doi:10.1615/JMachLearnModelComput.2020035155
[59] Taylor, J.; Benjamini, Y., RestrictedMVN: Multivariate normal restricted by affine constraints (2016)
[60] Wahl, F.; Espinasse, T. (2018)
[61] Wang, X.; Berger, J. O., Estimating shape constrained functions using Gaussian processes, SIAM/ASA Journal on Uncertainty Quantification, 4, 1, 1-25 (2016) · Zbl 1347.60040 · doi:10.1137/140955033
[62] Xuming, H.; Peide, S., Monotone B-spline smoothing, Journal of the American Statistical Association, 93, 643-50 (1996) · Zbl 1127.62322
[63] Zhang, J.; Lin, L., Bounded regression with Gaussian process projection (2018)
[64] Zhou, S.; Giulani, P.; Piekarewicz, J.; Bhattacharya, A.; Pati, D., Reexamining the proton-radius problem using constrained Gaussian processes, Physical Review C, 99, 5, 055202 (2019) · doi:10.1103/PhysRevC.99.055202
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.