Entropy-based closure for probabilistic learning on manifolds. (English) Zbl 1459.62239
Summary: In a recent paper, the authors proposed a general methodology for probabilistic learning on manifolds. The method was used to generate numerical samples that are statistically consistent with an existing dataset construed as a realization from a non-Gaussian random vector. The manifold structure is learned using diffusion manifolds and the statistical sample generation is accomplished using a projected Itô stochastic differential equation. This probabilistic learning approach has been extended to polynomial chaos representation of databases on manifolds and to probabilistic nonconvex constrained optimization with a fixed budget of function evaluations. The methodology introduces an isotropic-diffusion kernel with hyperparameter \({\epsilon}\). Currently, \({\epsilon}\) is more or less arbitrarily chosen. In this paper, we propose a selection criterion for identifying an optimal value of \({\epsilon}\), based on a maximum entropy argument. The result is a comprehensive, closed, probabilistic model for characterizing data sets with hidden constraints. This entropy argument ensures that out of all possible models, this is the one that is the most uncertain beyond any specified constraints, which is selected. Applications are presented for several databases.
MSC:
| 62R30 | Statistics on manifolds |
| 62H11 | Directional data; spatial statistics |
| 62-08 | Computational methods for problems pertaining to statistics |
| 65C05 | Monte Carlo methods |
| 65K10 | Numerical optimization and variational techniques |
| 68T05 | Learning and adaptive systems in artificial intelligence |
Keywords:
statistical learning; probabilistic learning; probability distribution on manifolds; Markov chain Monte Carlo generator; diffusion maps; entropy principleReferences:
| [1] | Soize, C.; Ghanem, R., Data-driven probability concentration and sampling on manifold, J. Comput. Phys., 321, 242-258 (2016) · Zbl 1349.62202 |
| [2] | Soize, C.; Ghanem, R., Polynomial chaos representation of databases on manifolds, J. Comput. Phys., 335, 201-221 (2017) · Zbl 1375.60028 |
| [3] | Schölkopf, B.; Smola, A.; Müller, K., Kernel principal component analysis, (Gerstner, W.; Germond, A.; Hasler, M.; Nicoud, J., Artificial Neural Networks ICANN’97. Artificial Neural Networks ICANN’97, Lecture Notes in Computer Science, vol. 1327 (1997), Springer: Springer Berlin, Heidelberg), 583-588 |
| [4] | Vapnik, V., The Nature of Statistical Learning Theory (2000), Springer: Springer New York · Zbl 0934.62009 |
| [5] | Aggarwal, C. C.; Zhai, C., Mining Text Data (2012), Springer Science & Business Media |
| [6] | Dalalyan, A. S.; Tsybakov, A. B., Sparse regression learning by aggregation and langevin monte-carlo, J. Comput. Syst. Sci., 78, 5, 1423-1443 (2012) · Zbl 1244.62054 |
| [7] | Murphy, K. P., Machine Learning: A Probabilistic Perspective (2012), MIT Press · Zbl 1295.68003 |
| [8] | Balcan, M.-F. F.; Feldman, V., Statistical active learning algorithms, (Advances in Neural Information Processing Systems (2013)), 1295-1303 |
| [9] | James, G.; Witten, D.; Hastie, T.; Tibshirani, R., An Introduction to Statistical Learning, vol. 112 (2013), Springer · Zbl 1281.62147 |
| [10] | Dong, X.; Gabrilovich, E.; Heitz, G.; Horn, W.; Lao, N.; Murphy, K.; Strohmann, T.; Sun, S.; Zhang, W., Knowledge vault: a web-scale approach to probabilistic knowledge fusion, (Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2014), ACM), 601-610 |
| [11] | Ghahramani, Z., Probabilistic machine learning and artificial intelligence, Nature, 521, 7553, 452 (2015) |
| [12] | Taylor, J.; Tibshirani, R. J., Statistical learning and selective inference, Proc. Natl. Acad. Sci., 112, 25, 7629-7634 (2015) · Zbl 1359.62228 |
| [13] | Ghanem, R.; Higdon, D.; Owhadi, H., Handbook of Uncertainty Quantification, vol. 1 to 3 (2017), Springer: Springer Cham, Switzerland · Zbl 1372.60001 |
| [14] | Jones, D.; Schonlau, M.; Welch, W., Efficient global optimization of expensive black-box functions, J. Glob. Optim., 13, 4, 455-492 (1998) · Zbl 0917.90270 |
| [15] | Queipo, N.; Haftka, R.; Shyy, W.; Goel, T.; Vaidyanathan, R.; Tucker, K., Surrogate-based analysis and optimization, Prog. Aerosp. Sci., 41, 1, 1-28 (2005) |
| [16] | Byrd, R.; Chin, G.; Neveitt, W.; Nocedal, J., On the use of stochastic Hessian information in optimization methods for machine learning, SIAM J. Optim., 21, 3, 977-995 (2011) · Zbl 1245.65062 |
| [17] | Homem-de Mello, T.; Bayraksan, G., Monte Carlo sampling-based methods for stochastic optimization, Surv. Oper. Res. Manag. Sci., 19, 1, 56-85 (2014) |
| [18] | Kleijnen, J.; van Beers, W.; van Nieuwenhuyse, I., Constrained optimization in expensive simulation: Novel approach, Eur. J. Oper. Res., 202, 1, 164-174 (2010) · Zbl 1189.90156 |
| [19] | Wang, Z.; Zoghi, M.; Hutter, F.; Matheson, D.; de Freitas, N., Bayesian optimization in a billion dimensions via random embeddings, J. Artif. Intell. Res., 55, 361-387 (2016) · Zbl 1358.90089 |
| [20] | Xie, J.; Frazier, P.; Chick, S., Bayesian optimization via simulation with pairwise sampling and correlated pair beliefs, Oper. Res., 64, 2, 542-559 (2016) · Zbl 1342.90107 |
| [21] | Du, X.; Chen, W., Sequential optimization and reliability assessment method for efficient probabilistic design, ASME J. Mech. Design, 126, 2, 225-233 (2004) |
| [22] | Eldred, M., Design under uncertainty employing stochastic expansion methods, Int. J. Uncertain. Quantificat., 1, 2, 119-146 (2011) · Zbl 1225.65064 |
| [23] | Yao, W.; Chen, X.; Luo, W.; vanTooren, M.; Guo, J., Review of uncertainty-based multidisciplinary design optimization methods for aerospace vehicles, Prog. Aerosp. Sci., 47, 450-479 (2011) |
| [24] | Ghanem, R.; Soize, C., Probabilistic nonconvex constrained optimization with fixed number of function evaluations, Int. J. Numer. Methods Eng., 113, 4, 719-741 (2018) · Zbl 07867277 |
| [25] | Ghanem, R.; Soize, C.; Thimmisetty, C.-R., Optimal well-placement using a probabilistic learning, Data-Enabled Discov. Appl., 2, 1, 4 (2018) |
| [26] | Thimmisetty, C.; Tsilifis, P.; Ghanem, R., Homogeneous chaos basis adaptation for design optimization under uncertainty: application to the oil well placement problem, Uncertainty Quantification for Engineering Design. Uncertainty Quantification for Engineering Design, Artif. Intell. Eng. Des. Anal. Manuf., 31, 3, 265-276 (2017) |
| [27] | Roesler, E.; Taylor, M.; Lin, W.; Tang, Q.; Klein, S., Climatology of the accelerated climate model for energy’s community atmospheric model, version 5, configured with variable resolution over the continental united states, Geosci. Model Dev. (2019), in preparation |
| [28] | Terai, C.; Caldwell, P.; Klein, S.; Tang, Q.; Branstetter, M., The atmospheric hydrologic cycle in the acme v0.3 model, Clim. Dyn., 50, 9-10, 3251-3279 (2018) |
| [29] | Soize, C.; Ghanem, R.; Safta, C.; Huan, X.; anf, Z. V.; Oefelein, J.; Lacaze, G.; Najm, H., Enhancing model predictability for a scramjet using probabilistic learning on manifold, AIAA J. (2018) |
| [30] | Bader, D.; Collins, W.; Jacob, R.; Jones, P.; Rasch, P.; Taylor, M.; Thornton, P.; Williams, D.; Bader, D.; Collins, W.; Jacob, R.; Jones, P.; Rasch, P.; Taylor, M.; Thornton, P.; Williams, D., Accelerated climate modeling for energy (acme) project strategy and initial implementation plan (2014), Tech. rep. |
| [31] | Soize, C., Polynomial chaos expansion of a multimodal random vector, SIAM/ASA J. Uncert. Quantif., 3, 1, 34-60 (2015) · Zbl 1327.62332 |
| [32] | Bowman, A.; Azzalini, A., Applied Smoothing Techniques for Data Analysis (1997), Oxford University Press: Oxford University Press Oxford, UK · Zbl 0889.62027 |
| [33] | Scott, D., Multivariate Density Estimation: Theory, Practice, and Visualization (2015), John Wiley and Sons: John Wiley and Sons New York · Zbl 1311.62004 |
| [34] | Soize, C., Construction of probability distributions in high dimension using the maximum entropy principle. applications to stochastic processes, random fields and random matrices, Int. J. Numer. Methods Eng., 76, 10, 1583-1611 (2008) · Zbl 1195.74311 |
| [35] | Girolami, M.; Calderhead, B., Riemann manifold Langevin and Hamiltonian Monte Carlo methods, J. R. Stat. Soc., 73, 2, 123-214 (2011) · Zbl 1411.62071 |
| [36] | Neal, R., MCMC using hamiltonian dynamics, (Brooks, S.; Gelman, A.; Jones, G.; Meng, X.-L., Handbook of Markov Chain Monte Carlo (2011), Chapman and Hall-CRC Press: Chapman and Hall-CRC Press Boca Raton), Ch. 5 · Zbl 1229.65018 |
| [37] | Spall, J., Introduction to Stochastic Search and Optimization (2003), Wiley-Interscience · Zbl 1088.90002 |
| [38] | Coifman, R.; Lafon, S.; Lee, A.; Maggioni, M.; Nadler, B.; Warner, F.; Zucker, S., Geometric diffusions as a tool for harmonic analysis and structure definition of data: diffusion maps, Proc. Natl. Acad. Sci., 102, 21, 7426-7431 (2005) · Zbl 1405.42043 |
| [39] | Coifman, R.; Lafon, S., Diffusion maps, applied and computational harmonic analysis, Appl. Comput. Harmon. Anal., 21, 1, 5-30 (2006) · Zbl 1095.68094 |
| [40] | Jaynes, E. T., Information theory and statistical mechanics, Phys. Rev., 106, 4, 620-630 (1957) · Zbl 0084.43701 |
| [41] | Jaynes, E. T., Information theory and statistical mechanics, Phys. Rev., 108, 2, 171-190 (1957) · Zbl 0084.43701 |
| [42] | Gray, R. M., Entropy and Information Theory (2011), Springer: Springer New York · Zbl 1216.94001 |
| [43] | Soize, C., Uncertainty Quantification. An Accelerated Course with Advanced Applications in Computational Engineering (2017), Springer: Springer New York · Zbl 1377.60002 |
| [44] | Shannon, C. E., A mathematical theory of communication, Bell Syst. Tech. J., 27, 379-423 (1948), and 623-659 · Zbl 1154.94303 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
