×

Diffusion maps-aided neural networks for the solution of parametrized PDEs. (English) Zbl 1506.65180

Summary: This work introduces a surrogate modeling strategy, based on diffusion maps manifold learning and artificial neural networks. On this basis, a numerical procedure is developed for cost-efficient predictions of a complex system’s response modeled by parametrized partial differential equations. The idea is to utilize a collection of solution snapshots, obtained by solving the partial differential equation for a small number of parameter values, in order to establish an efficient yet accurate mapping from the problem’s parametric space to its solution space. In common practice, solving the partial differential equations in the framework of the finite element method leads to high-dimensional data sets, which are a major challenge for machine learning algorithms to handle (curse of dimensionality). To overcome this problem, the proposed method exploits the dimensionality reduction properties of the diffusion maps algorithm in order to obtain a meaningful low-dimensional representation of the solution data set. With this approach, a reduced set of ‘hyperparameters’ is obtained, namely, the diffusion map coordinates, that characterize the high-dimensional solution vectors. Using this reduced representation, a feed-forward neural network is efficiently trained that maps the problem’s parameter values to their images in the low-dimensional diffusion maps space. This approach offers two advantages. The obvious one is that it reduces the computational resources and CPU time needed to train the neural network, which can be prohibitive for high dimensional problems. The second advantage is that training the neural network on the diffusion map coordinates, essentially translates to using the diffusion maps distance in the MSE loss function of the network. Compared to the Euclidean distance in the ambient space, the diffusion distance gives a better approximation of the distance between two points on the solution manifold, which leads to a more accurate network. Lastly, a mapping is developed based on the Laplacian pyramids scheme in order to convert points from the diffusion maps space back to the solution space. The composition of the neural network with the Laplacian pyramid scheme, substitutes the process of solving the partial differential equation under consideration, and is capable of providing the additional system solutions at a very low cost and high accuracy.

MSC:

65M99 Numerical methods for partial differential equations, initial value and time-dependent initial-boundary value problems
68T05 Learning and adaptive systems in artificial intelligence
Full Text: DOI

References:

[1] Katrakazas, C.; Quddus, M.; Chen, W.-H.; Deka, L., Real-time motion planning methods for autonomous on-road driving: State-of-the-art and future research directions, Transp. Res. C, 60, 416-442 (2015)
[2] Lin, P. Y.; Chung, L. L.; Loh, C. H., Semiactive control of building structures with semiactive tuned mass damper, Comput.-Aided Civ. Infrastruct. Eng., 20, 1, 35-51 (2005)
[3] Lucia, D. J.; Beran, P. S.; Silva, W. A., Reduced-order modeling: new approaches for computational physics, Prog. Aerosp. Sci., 40, 1, 51-117 (2004)
[4] O. Ezvan, A. Batou, C. Soize, L. Gagliardini, Multilevel model reduction for uncertainty quantification in computational structural dynamics, Comput. Mech. 59, http://dx.doi.org/10.1007/s00466-016-1348-1. · Zbl 1398.74327
[5] Jensen, H.; Munoz, A.; Papadimitriou, C.; Millas, E., Model-reduction techniques for reliability-based design problems of complex structural systems, Reliab. Eng. Syst. Saf., 149, 204-217 (2016)
[6] Mignolet, M. P.; Przekop, A.; Rizzi, S. A.; Spottswood, S. M., A review of indirect/non-intrusive reduced order modeling of nonlinear geometric structures, J. Sound Vib., 332, 10, 2437-2460 (2013)
[7] Rathinam, M.; Petzold, L. R., A new look at proper orthogonal decomposition, SIAM J. Numer. Anal., 41, 5, 1893-1925 (2003) · Zbl 1053.65106
[8] Amsallem, D.; Farhat, C., Interpolation method for adapting reduced-order models and application to aeroelasticity, AIAA J., 46, 7, 1803-1813 (2008)
[9] Farhat, C.; Chapman, T.; Avery, P., Structure-preserving, stability, and accuracy properties of the energy-conserving sampling and weighting method for the hyper reduction of nonlinear finite element dynamic models, Internat. J. Numer. Methods Engrg., 102, 5, 1077-1110 (2015) · Zbl 1352.74349
[10] Proper orthogonal decomposition and its applications—part i: Theory, J. Sound Vib., 252, 3, 527-544 (2002) · Zbl 1237.65040
[11] T. Lieu, M. Lesoinne, Parameter adaptation of reduced order models for three-dimensional flutter analysis, in: 42nd AIAA Aerospace Sciences Meeting and Exhibit, http://dx.doi.org/10.2514/6.2004-888.
[12] Lieu, T.; Farhat, C.; Lesoinne, M., Reduced-order fluid/structure modeling of a complete aircraft configuration, Comput. Methods Appl. Mech. Engrg., 195, 41, 5730-5742 (2006) · Zbl 1124.76042
[13] Amsallem, D.; Zahr, M. J.; Farhat, C., Nonlinear model order reduction based on local reduced-order bases, Internat. J. Numer. Methods Engrg., 92, 10, 891-916 (2012) · Zbl 1352.65212
[14] Amsallem, D.; Zahr, M.; Washabaugh, K., Fast local reduced basis updates for the efficient reduction of nonlinear systems with hyper-reduction, Adv. Comput. Math., 41, 1187-1230 (2015) · Zbl 1331.65094
[15] Li, H.; Lu, Z.; Yue, Z., Support vector machine for structural reliability analysis, Appl. Math. Mech., 27, 1295-1303 (2006) · Zbl 1167.62469
[16] Tiago, C.; Leitao, V., Application of radial basis functions to linear and nonlinear structural analysis problems, Comput. Math. Appl., 51, 8, 1311-1334 (2006), radial Basis Functions and Related Multivariate Meshfree Approximation Methods: Theory and Applications · Zbl 1148.41031
[17] Su, G.; Peng, L.; Hu, L., A gaussian process-based dynamic surrogate model for complex engineering structural reliability analysis, Struct. Saf., 68, 97-109 (2017)
[18] Schmidhuber, J., Deep learning in neural networks: An overview, Neural Netw., 61, 85-117 (2015)
[19] Papadrakakis, M.; Papadopoulos, V.; Lagaros, N. D., Structural reliability analyis of elastic-plastic structures using neural networks and monte carlo simulation, Comput. Methods Appl. Mech. Engrg., 136, 1, 145-163 (1996) · Zbl 0893.73079
[20] Hurtado, J., Neural networks in stochastic mechanics, Arch. Comput. Methods Eng., 8, 303-342 (2001) · Zbl 1043.74053
[21] Mohajerin, N.; Waslander, S. L., Multistep prediction of dynamic systems with recurrent neural networks, IEEE Trans. Neural Netw. Learn. Syst., 30, 11, 3370-3383 (2019)
[22] Vlachas, P. R.; Byeon, W.; Wan, Z. Y.; Sapsis, T. P.; Koumoutsakos, P., Data-driven forecasting of high-dimensional chaotic systems with long short-term memory networks, Proc. R. Soc. A, 474, 2213, Article 20170844 pp. (2018) · Zbl 1402.92030
[23] Wu, R.-T.; Jahanshahi, M. R., Deep convolutional neural network for structural dynamic response estimation and system identification, J. Eng. Mech., 145, 1, Article 04018125 pp. (2019)
[24] S. Vijayanarasimhan, J. Shlens, R. Monga, J. Yagnik, Deep networks with large output spaces, in: CoRR abs/1412.7479.
[25] Krizhevsky, A.; Sutskever, I.; Hinton, G. E., Imagenet classification with deep convolutional neural networks, Commun. ACM, 60, 6, 84-90 (2017)
[26] Brunton, S. L.; Noack, B. R.; Koumoutsakos, P., Machine learning for fluid mechanics, Annu. Rev. Fluid Mech., 52, 1, 477-508 (2020) · Zbl 1439.76138
[27] Hesthaven, J.; Ubbiali, S., Non-intrusive reduced order modeling of nonlinear problems using neural networks, J. Comput. Phys., 363, 55-78 (2018) · Zbl 1398.65330
[28] Park, K. H.; Jun, S. O.; Baek, S. M.; Cho, M. H.; Yee, K. J.; Lee, D. H., Reduced-order model with an artificial neural network for aerostructural design optimization, J. Aircr., 50, 4, 1106-1116 (2013)
[29] Guo, M.; Hesthaven, J. S., Reduced order modeling for nonlinear structural analysis using gaussian process regression, Comput. Methods Appl. Mech. Engrg., 341, 807-826 (2018) · Zbl 1440.65206
[30] Xiao, D.; Fang, F.; Pain, C.; Navon, I., A parameterized non-intrusive reduced order model and error analysis for general time-dependent nonlinear partial differential equations and its applications, Comput. Methods Appl. Mech. Engrg., 317, 868-889 (2017) · Zbl 1439.65124
[31] Peherstorfer, B.; Willcox, K., Data-driven operator inference for nonintrusive projection-based model reduction, Comput. Methods Appl. Mech. Engrg., 306, 196-215 (2016) · Zbl 1436.93062
[32] Giovanis, D.; Shields, M., Uncertainty quantification for complex systems with very high dimensional response using grassmann manifold variations, J. Comput. Phys., 364, 393-415 (2018) · Zbl 1392.65014
[33] C. Lataniotis, S. Marelli, B. Sudret, Extending classical surrogate modelling to ultrahigh dimensional problems through supervised dimensionality reduction: a data-driven approach, arXiv:abs/1812.06309.
[34] Schölkopf, B.; Smola, A.; Müller, K.-R., Nonlinear component analysis as a kernel eigenvalue problem, Neural Comput., 10, 5, 1299-1319 (1998)
[35] Donoho, D. L.; Grimes, C., Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data, Proc. Natl. Acad. Sci., 100, 10, 5591-5596 (2003) · Zbl 1130.62337
[36] Belkin, M.; Niyogi, P., Laplacian eigenmaps for dimensionality reduction and data representation, Neural Comput., 15, 6, 1373-1396 (2003) · Zbl 1085.68119
[37] Zhang, Z.; Zha, H., Principal manifolds and nonlinear dimensionality reduction via tangent space alignment, SIAM J. Sci. Comput., 26, 1, 313-338 (2004) · Zbl 1077.65042
[38] Coifman, R. R.; Lafon, S.; Lee, A. B.; Maggioni, M.; Nadler, B.; Warner, F.; Zucker, S. W., Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps, Proc. Natl. Acad. Sci., 102, 21, 7426-7431 (2005) · Zbl 1405.42043
[39] B. Sonday, A. Singer, C. Gear, I.G. Kevrekidis, Manifold learning techniques and model reduction applied to dissipative pdes.
[40] C.A. Thimmisetty, R.G. Ghanem, J.A. White, X. Chen, High-dimensional intrinsic interpolation using gaussian process regression and diffusion maps, Math. Geosci. 50 (1), http://dx.doi.org/10.1007/s11004-017-9705-y. · Zbl 1406.86047
[41] Xing, W.; Triantafyllidis, V.; Shah, A.; Nair, P.; Zabaras, N., Manifold learning for the emulation of spatial fields from computational models, J. Comput. Phys., 326, 666-690 (2016) · Zbl 1373.68340
[42] Soize, C.; Ghanem, R., Data-driven probability concentration and sampling on manifold, J. Comput. Phys., 321, 242-258 (2016), URL http://www.sciencedirect.com/science/article/pii/S0021999116301899 · Zbl 1349.62202
[43] Mishne, G.; Shaham, U.; Cloninger, A.; Cohen, I., Diffusion nets, Appl. Comput. Harmon. Anal., 47, 2, 259-285 (2019) · Zbl 1430.68278
[44] Kalogeris, I.; Papadopoulos, V., Diffusion maps-based surrogate modeling: An alternative machine learning approach, Internat. J. Numer. Methods Engrg., 121, 4, 602-620 (2020) · Zbl 07843214
[45] Berry, T.; Harlim, J., Variable bandwidth diffusion kernels, Appl. Comput. Harmon. Anal., 40, 1, 68-96 (2016) · Zbl 1343.94020
[46] Coifman, R. R.; Lafon, S., Diffusion maps, Appl. Comput. Harmon. Anal., 21, 1, 5-30 (2006), special Issue: Diffusion Maps and Wavelets · Zbl 1095.68094
[47] Coifman, R. R.; Kevrekidis, I. G.; Lafon, S.; Maggioni, M.; Nadler, B., Diffusion maps reduction coordinates and low dimensional representation of stochastic systems, Multiscale Model. Simul., 7, 2, 842-864 (2008) · Zbl 1175.60058
[48] Soize, C.; Ghanem, R., Physics-constrained non-gaussian probabilistic learning on manifolds, Internat. J. Numer. Methods Engrg., 121, 1, 110-145 (2020) · Zbl 07841255
[49] Coifman, R. R.; Hirn, M. J., Diffusion maps for changing data, Appl. Comput. Harmon. Anal., 36, 1, 79-107 (2014), URL http://www.sciencedirect.com/science/article/pii/S1063520313000225 · Zbl 1302.62122
[50] Chen, Y. F.; Liu, S.; Liu, M.; Miller, J.; How, J. P., Motion planning with diffusion maps, (2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2016)), 1423-1430
[51] Sain, S. R.; Scott, D. W., On locally adaptive density estimation, J. Amer. Statist. Assoc., 91, 436, 1525-1534 (1996) · Zbl 0882.62035
[52] Terrell, G. R.; Scott, D. W., Variable kernel density estimation, Ann. Statist., 20, 3, 1236-1265 (1992) · Zbl 0763.62024
[53] Erban, R.; Frewen, T. A.; Wang, X.; Elston, T. C.; Coifman, R.; Nadler, B.; Kevrekidis, I. G., Variable-free exploration of stochastic models: A gene regulatory network example, J. Chem. Phys., 126, 15, Article 155103 pp. (2007)
[54] Belkin, M.; Niyogi, P.; Sindhwani, V., Manifold regularization: A geometric framework for learning from labeled and unlabeled examples, J. Mach. Learn. Res., 7, 2434 (2006) · Zbl 1222.68144
[55] Coifman, R. R.; Lafon, S., Geometric harmonics: A novel tool for multiscale out-of-sample extension of empirical functions, Appl. Comput. Harmon. Anal., 21, 1, 31-52 (2006), special Issue: Diffusion Maps and Wavelets · Zbl 1095.68095
[56] Powell, M. J.D., Algorithms for approximation, 143-167 (1987) · Zbl 0638.41001
[57] Burt, P.; Adelson, E., The laplacian pyramid as a compact image code, IEEE Trans. Commun., 31, 4, 532-540 (1983)
[58] Bentley, J. L., Multidimensional binary search trees used for associative searching, Commun. ACM, 18, 9, 509-517 (1975) · Zbl 0306.68061
[59] Holden, D.; Duong, B. C.; Datta, S.; Nowrouzezahrai, D., Subspace neural physics: Fast data-driven interactive simulation, (Proceedings of the 18th Annual ACM SIGGRAPH/Eurographics Symposium on Computer Animation, SCA ’19 (2019), Association for Computing Machinery: Association for Computing Machinery New York, NY, USA)
[60] MacQueen, J. B., Some methods for classification and analysis of multivariate observations (1967) · Zbl 0214.46201
[61] Faraoun, K. M.; Boukelif, A., Neural networks learning improvement using the k-means clustering algorithm to detect network intrusions, INFOCOMP J. Comput. Sci., 5, 3, 28-36 (2006)
[62] Basdevant, C.; Deville, M.; Haldenwang, P.; Lacroix, J.; Ouazzani, J.; Peyret, R.; Orlandi, P.; Patera, A., Spectral and finite difference solutions of the burgers equation, Comput. & Fluids, 14, 1, 23-41 (1986) · Zbl 0612.76031
[63] Marconi, J.; Tiso, P.; Braghin, F., A nonlinear reduced order model with parametrized shape defects, Comput. Methods Appl. Mech. Engrg., 360, Article 112785 pp. (2020), URL http://www.sciencedirect.com/science/article/pii/S0045782519306772 · Zbl 1441.74262
[64] Kast, M.; Guo, M.; Hesthaven, J. S., A non-intrusive multifidelity method for the reduced order modeling of nonlinear problems, Comput. Methods Appl. Mech. Engrg., 364, Article 112947 pp. (2020) · Zbl 1442.65094
[65] Goodfellow, I.; Bengio, Y.; Courville, A., Deep Learning (2016), The MIT Press · Zbl 1373.68009
[66] Géron, A., Hands-On Machine Learning with Scikit-Learn and Tensorflow: Concepts, Tools, and Techniques to Build Intelligent Systems (2017), O’Reilly Media, Inc.
[67] Hagan, M. T.; Menhaj, M. B., Training feedforward networks with the marquardt algorithm, IEEE Trans. Neural Netw., 5, 6, 989-993 (1994)
[68] D.P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: CoRR abs/1412.6980.
[69] Cybenko, G., Approximation by superpositions of a sigmoidal function, Math. Control Signal Syst., 2, 303-314 (1989) · Zbl 0679.94019
[70] Cybenko, G., Continuous Valued Neural Networks with Two Hidden Layers are SufficientTech. rep. (1988), Department of Computer Science, Tufts University
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.