Time-series machine-learning error models for approximate solutions to parameterized dynamical systems. (English) Zbl 1442.65446
Summary: This work proposes a machine-learning framework for modeling the error incurred by approximate solutions to parameterized dynamical systems. In particular, we extend the machine-learning error models (MLEM) framework proposed in [B. A. Freno and the second author, ibid. 348, 250–296 (2019; Zbl 1440.65058)] to dynamical systems. The proposed Time-Series Machine-Learning Error Modeling (T-MLEM) method constructs a regression model that maps features – which comprise error indicators that are derived from standard a posteriori error-quantification techniques – to a random variable for the approximate-solution error at each time instance. The proposed framework considers a wide range of candidate features, regression methods, and additive noise models. We consider primarily recursive regression techniques developed for time-series modeling, including both classical time-series models (e.g., autoregressive models) and recurrent neural networks (RNNs), but also analyze standard non-recursive regression techniques (e.g., feed-forward neural networks) for comparative purposes. Numerical experiments conducted on multiple benchmark problems illustrate that the long short-term memory (LSTM) neural network, which is a type of RNN, outperforms other methods and yields substantial improvements in error predictions over traditional approaches.
MSC:
| 65P99 | Numerical problems in dynamical systems |
| 70-08 | Computational methods for problems pertaining to mechanics of particles and systems |
| 62M10 | Time series, auto-correlation, regression, etc. in statistics (GARCH) |
| 62H30 | Classification and discrimination; cluster analysis (statistical aspects) |
Keywords:
error modeling; time-series modeling; recurrent neural networks; long short-term memory networks; parameterized dynamical systems; model reductionCitations:
Zbl 1440.65058References:
| [1] | Babuška, I.; Rheinboldt, W., Error estimates for adaptive finite element computations, SIAM J. Numer. Anal., 15, 4, 736-754 (1978) · Zbl 0398.65069 |
| [2] | Babuška, I.; Rheinboldt, W., A-posteriori error estimates for the finite element method, Internat. J. Numer. Methods Engrg., 12, 10, 1597-1615 (1978) · Zbl 0396.65068 |
| [3] | Ainsworth, M.; Oden, J. T., A posteriori error estimation in finite element analysis, Comput. Methods Appl. Mech. Engrg., 142, 1-88 (1997) · Zbl 0895.76040 |
| [4] | Paraschivoiu, M.; Peraire, J.; Patera, A. T., A posteriori finite element bounds for linear-functional outputs of elliptic partial differential equations, Comput. Methods Appl. Mech. Engrg., 150, 1-4, 289-312 (1997) · Zbl 0907.65102 |
| [5] | Paraschivoiu, M.; Patera, A. T., A hierarchical duality approach to bounds for the outputs of partial differential equations, Comput. Methods Appl. Mech. Engrg., 158, 3-4, 389-407 (1998) · Zbl 0953.76054 |
| [6] | Maday, Y.; Patera, A. T., Numerical analysis of a posteriori finite element bounds for linear functional outputs, Math. Models Methods Appl. Sci., 10, 5, 785-799 (2000) · Zbl 1012.65109 |
| [7] | C. Prud’Homme, D.V. Rovas, K. Veroy, L. Machiels, Y. Maday, A. Patera, G. Turinici, Reduced-basis output bound methods for parametrized partial differential equations, in: Proceedings SMA Symposium, 2002. |
| [8] | Prud’Homme, C.; Rovas, D.; Veroy, K.; Machiels, L.; Maday, Y.; Patera, A.; Turinici, G., Reliable real-time solution of parametrized partial differential equations: Reduced-basis output bound methods, J. Fluids Eng., 124, 1, 70-80 (2001) |
| [9] | Grepl, M.; Patera, A., A posteriori error bounds for reduced-basis approximations of parametrized parabolic partial differential equations, ESAIM, 39, 1, 157-181 (2005) · Zbl 1079.65096 |
| [10] | Rovas, D.; Machiels, L.; Maday, Y., Reduced-basis output bound methods for parabolic problems, IMA J. Numer. Anal., 26, 423-445 (2006) · Zbl 1101.65099 |
| [11] | Homescu, C.; Petzold, L. R.; Serban, R., Error estimation for reduced-order models of dynamical systems, SIAM Rev., 49, 2 (2007) · Zbl 1117.65104 |
| [12] | Carlberg, K.; Barone, M.; Antil, H., Galerkin v. least-squares Petrov-Galerkin projection in nonlinear model reduction, J. Comput. Phys., 330, 693-734 (2017) · Zbl 1378.65145 |
| [13] | Bui-Thanh, T.; Wilcox, K.; Ghattas, O., Parametric reduced-order models for probabilistic analysis of unsteady aerodynamic applications, AIAA J., 46, 10, 2520-2529 (2008) |
| [14] | Bui-Thanh, T.; Willcox, K.; Ghattas, O., Model reduction for large-scale systems with high-dimensional parametric input space, SIAM J. Sci. Comput., 30, 6, 3270-3288 (2008) · Zbl 1196.37127 |
| [15] | Hinze, M.; Kunkel, M., Residual based sampling in POD model order reduction of drift-diffusion equations in parametrized electrical networks, ZAMM Z. Angew. Math. Mech., 92, 2, 91-104 (2012) · Zbl 1237.78038 |
| [16] | Wu, Y.; Hetmaniuk, U., Adaptive training of local reduced bases for unsteady incompressible Navier-Stokes flows, Internat. J. Numer. Methods Engrg., 103, 3, 183-204 (2015) · Zbl 1352.76070 |
| [17] | Yano, M.; Patera, A. T., An LP empirical quadrature procedure for reduced basis treatment of parametrized nonlinear PDEs, Comput. Methods Appl. Mech. Engrg. (2018) |
| [18] | Zahr, M. J.; Farhat, C., Progressive construction of a parametric reduced-order model for pde-constrained optimization, Internat. J. Numer. Methods Engrg., 102, 5, 1111-1135 (2015) · Zbl 1352.49029 |
| [19] | Zahr, M., Adaptive Model Reduction to Accelerate Optimization Problems Governed By Partial Differential Equations (2016), Stanford University, (Ph.D. thesis) |
| [20] | Zahr, M.; Carlberg, K.; Kouri, D., An efficient, globally convergent method for optimization under uncertainty using adaptive model reduction and sparse grids, SIAM/ASA J. Uncertain. Quantif., 6, 3, 877-912 (2019) · Zbl 1448.65057 |
| [21] | Lu, J. C.-C., An a posteriori Error Control Framework for Adaptive Precision Optimization using Discontinuous Galerkin Finite Element Method (2005), Massachusetts Institute of Technology, (Ph.D. thesis) |
| [22] | Ackmann, J.; Moarotzke, J.; Korn, P., Stochastic goal-oriented error estimation with memory, J. Comput. Phys., 348, 1, 195-219 (2017) · Zbl 1422.76131 |
| [23] | Venditti, D. A.; Darmofal, D. L., Adjoint error estimation and grid adaptation for functional outputs: application to quasi-one-dimensional flow, J. Comput. Phys., 164, 1, 204-227 (2000) · Zbl 0995.76057 |
| [24] | Venditti, D. A.; Darmofal, D. L., Grid adaptation for functional outputs: Application to two-dimensional inviscid flows, J. Comput. Phys., 176, 1, 40-69 (2002) · Zbl 1120.76342 |
| [25] | Blonigan, P. J., Least Squares Shadowing for Sensitivity Analysis of Large Chaotic Systems and Fluid Flows (2016), Massachusetts Institute of Technology, (Ph.D. thesis) |
| [26] | Y.S. Shimizu, Output-based error estimation for chaotic flows using reduced-order modeling, in: AIAA SciTech Forum, AIAA 2018-0826, , 2018. |
| [27] | Kennedy, M. C.; O’Hagan, A., Bayesian Calibration of computer models, J. R. Stat. Soc. Ser. B Stat. Methodol., 63, 3, 425-464 (2001) · Zbl 1007.62021 |
| [28] | Huang, D.; Allen, T. T.; Notz, W. I.; Miller, R. A., Sequential kriging optimization using multiple-fidelity evaluations, Struct. Multidiscip. Optim., 32, 5 (2006) |
| [29] | Eldred, M. S.; Giunta, A. A.; Collis, S. S.; Alexandrov, N. A.; Lewis, R. M., Second-order corrections for surrogate-based optimization with model hierarchies, (10th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference (2004), American Institute of Aeronautics and Astronautics) |
| [30] | March, A.; Willcox, K., Provably convergent multifidelity optimization algorithm not requiring high-fidelity derivatives, AIAA J., 50, 5 (2012) |
| [31] | Gano, S. E.; Renaud, J. E.; Sanders, B., Hybrid variable fidelity optimization by using a kriging-based scaling function, AIAA J., 43, 11 (2005) |
| [32] | Moosavi, A.; Ştefănescu, R.; Sandu, A., Multivariate predictions of local reduced-order-model errors and dimensions, Internat. J. Numer. Methods Engrg., 113, 3, 512-533 (2018) · Zbl 07874306 |
| [33] | Ştefănescu, R.; Moosavi, A.; Sandu, A., Parametric domain decomposition for accurate reduced order models: Applications of MP-LROM methodology, J. Comput. Appl. Math., 340, 629-644 (2018) · Zbl 1433.65186 |
| [34] | Drohmann, M.; Carlberg, K., The ROMES method for statistical modeling of reduced-order-model error, SIAM/ASA J. Uncertain. Quantif., 3, 116-145 (2015) · Zbl 1322.65029 |
| [35] | Pagani, S.; Manzoni, A.; Carlberg, K., Statistical closure modeling for reduced-order models of stationary systems by the ROMES method (2019) · Zbl 1498.65013 |
| [36] | Freno, B.; Carlberg, K., Machine-learning error models for approximate solutions to parameterized systems of nonlinear equations, Comput. Methods Appl. Mech. Engrg., 348, 250-296 (2019) · Zbl 1440.65058 |
| [37] | Trehan, S.; Carlberg, K.; Durlofsky, L. J., Error estimation for surrogate models of dynamical systems using machine learning, Internat. J. Numer. Methods Engrg., 112, 12, 1801-1827 (2017) · Zbl 07867191 |
| [38] | Pagani, S.; Manzoni, A.; Quarteroni, A., Efficient state/parameter estimation in nonlinear unsteady PDEs by a reduced basis ensemble Kalman filter, SIAM/ASA J. Uncertain. Quantif., 5, 890-921 (2017) · Zbl 1398.65280 |
| [39] | Rumelhart, D. E.; Hinton, G. E.; Williams, R. J., Learning representations by back-propagating errors, Nature, 323, 6088, 533-536 (1986) · Zbl 1369.68284 |
| [40] | Berkooz, G.; Holmes, P.; Lumley, J. L., The proper orthogonal decomposition in the analysis of turbulent flows, Annu. Rev. Fluid Mech., 25, 539-575 (1993) |
| [41] | Lee, K.; Carlberg, K. T., Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders, J. Comput. Phys., 404, Article 108973 pp. (2020) · Zbl 1454.65184 |
| [42] | Huynh, D.; Knezevic, D.; Chen, Y.; Hesthaven, J. S.; Patera, A., A natural-norm successive constraint method for inf-sup lower bounds, Comput. Methods Appl. Mech. Engrg., 199, 29-32, 1963-1975 (2010) · Zbl 1231.76208 |
| [43] | Everson, R.; Sirovich, L., Karhunen-Loève procedure for gappy data, J. Opt. Soc. Amer. A, 12 (1995), 1657-1644 |
| [44] | Drmac, Z.; Gugercin, S., A new selection operator for the discrete empirical interpolation method—improved a priori error bound and extensions, SIAM J. Sci. Comput., 38, A631-A648 (2016) · Zbl 1382.65193 |
| [45] | Hochreiter, S.; Schmidhuber, J., Long short-term memory, Neural Comput., 9, 8, 1735-1780 (1997) |
| [46] | Olah, C., Understanding LSTM networks (2015) |
| [47] | Mozer, M. C., A focused backpropagation algorithm for temporal pattern recognition, Complex Syst., 3 (1989) · Zbl 0727.68101 |
| [48] | Robinson, A. J.; Fallside, F., The Utility Driven Dynamic Error Propagation NetworkTech. Rep. CUED/F-INFENG/TR.1 (1987), Engineering Department, Cambridge University: Engineering Department, Cambridge University Cambridge, UK |
| [49] | Goodfellow, I.; Bengio, Y.; Courville, A., Deep Learning (2016), MIT Press, http://www.deeplearningbook.org · Zbl 1373.68009 |
| [50] | Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; Vanderplas, J.; Passos, A.; Cournapeau, D.; Brucher, M.; Perrot, M.; Duchesnay, E., Scikit-learn: Machine learning in Python, J. Mach. Learn. Res., 12, 2825-2830 (2011) · Zbl 1280.68189 |
| [51] | Chollet, F., Keras (2015) |
| [52] | Kingma, D. P.; Ba, J., Adam: A method for stochastic optimization, CoRR, abs/1412.6980 (2015) |
| [53] | Rasmussen, C. E.; Williams, C. K.I., Gaussian Processes for Machine Learning (2006), The MIT Press · Zbl 1177.68165 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
