Learning theory for dynamical systems. (English) Zbl 1526.37089
Summary: The task of modeling and forecasting a dynamical system is one of the oldest problems, and it remains challenging. Broadly, this task has two subtasks: extracting the full dynamical information from a partial observation, and then explicitly learning the dynamics from this information. We present a mathematical framework in which the dynamical information is represented in the form of an embedding. The framework combines the two subtasks using the language of spaces, maps, and commutations. The framework also unifies two of the most common learning paradigms: delay-coordinates and reservoir computing. We use this framework as a platform for two other investigations of the reconstructed system, its dynamical stability and the growth of error under iterations. We show that these questions are deeply tied to more fundamental properties of the underlying system, i.e., the behavior of matrix cocycles over the base dynamics, its nonuniform hyperbolic behavior, and its decay of correlations. Thus, our framework bridges the gap between universally observed behavior of dynamics modeling and the spectral, differential, and ergodic properties intrinsic to the dynamics.
MSC:
| 37M25 | Computational methods for ergodic theory (approximation of invariant measures, computation of Lyapunov exponents, entropy, etc.) |
| 37M05 | Simulation of dynamical systems |
| 37M10 | Time series analysis of dynamical systems |
| 37N30 | Dynamical systems in numerical analysis |
| 37D25 | Nonuniformly hyperbolic systems (Lyapunov exponents, Pesin theory, etc.) |
| 37D20 | Uniformly hyperbolic systems (expanding, Anosov, Axiom A, etc.) |
Keywords:
matrix cocycle; Lyapunov exponent; reservoir computing; delay-coordinates; mixing; direct forecast; iterative forecastReferences:
| [1] | Alexander, R. and Giannakis, D., Operator-theoretic framework for forecasting nonlinear time series with kernel analog techniques, Phys. D, 409 (2020), 132520, doi:10.1016/j.physd.2020.132520. · Zbl 1496.37085 |
| [2] | Alves, J., Luzzatto, S., and Pinheiro, V., Markov structures and decay of correlations for non-uniformly expanding dynamical systems, Ann. Inst. H. Poincaré C Non Linéaire, 22 (2005), pp. 817-839, doi:10.1016/j.anihpc.2004.12.002. · Zbl 1134.37326 |
| [3] | Alves, J., Luzzatto, S., and Pinheiro, V., Lyapunov exponents and rates of mixing for one-dimensional maps, Ergodic Theory Dynam. Systems, 24 (2004), pp. 637-657, doi:10.1017/S0143385703000579. · Zbl 1049.37025 |
| [4] | Arnold, L., Random Dynamical Systems, Springer, 1991, doi:10.1007/978-3-662-12878-7. · Zbl 0736.58029 |
| [5] | Barreira, L. and Valls, C., Smoothness of invariant manifolds for nonautonomous equations, Comm. Math. Phys., 259 (2005), pp. 639-677, doi:10.1007/s00220-005-1380-z. · Zbl 1098.34041 |
| [6] | Barreira, L. and Valls, C., Stable manifolds for nonautonomous equations without exponential dichotomy, J. Differential Equations, 221 (2006), pp. 58-90, doi:10.1016/j.jde.2005.04.005. · Zbl 1098.34036 |
| [7] | Berry, T., J. Cressman, R., Gregurić-Ferenček, Z., and Sauer, T., Time-scale separation from diffusion-mapped delay coordinates, SIAM J. Appl. Dyn. Syst., 12 (2013), pp. 618-649, doi:10.1137/12088183x. · Zbl 1291.37101 |
| [8] | Berry, T., Giannakis, D., and Harlim, J., Nonparametric forecasting of low-dimensional dynamical systems, Phys. Rev. E, 91 (2015), 032915. doi:10.1103/PhysRevE.91.032915. |
| [9] | Berry, T. and Harlim, J., Correcting biased observation model error in data assimilation, Monthly Weather Rev., 145 (2017), pp. 2833-2853, doi:10.1175/MWR-D-16-0428.1. |
| [10] | Blumenthal, A. and Young, L., Equivalence of physical and SRB measures in random dynamical systems, Nonlinearity, 32 (2019), pp. 1494-1524, doi:10.1088/1361-6544/aafaa8. · Zbl 1423.37023 |
| [11] | Bochi, J. and Viana, M., The Lyapunov exponents of generic volume-preserving and symplectic maps, Ann. of Math. (2), 161 (2005), pp. 1423-1485, doi:10.4007/annals.2005.161.1423. · Zbl 1101.37039 |
| [12] | Bollt, E., On explaining the surprising success of reservoir computing forecaster of chaos? The universal machine learning dynamical system with contrast to VAR and DMD, Chaos, 31 (2021), 013108, doi:10.1063/5.0024890. · Zbl 1512.68266 |
| [13] | Box, G., Jenkins, G., Reinsel, G., and Ljung, G., Time Series Analysis: Forecasting and Control, John Wiley & Sons, 2015. · Zbl 1317.62001 |
| [14] | Bramburger, J., Brunton, S., and Kutz, J., Deep learning of conjugate mappings, Phys. D, 427 (2021), 133008, doi:10.1016/j.physd.2021.133008. · Zbl 1491.37074 |
| [15] | Casdagli, M., Nonlinear prediction of chaotic time series, Phys. D, 35 (1989), pp. 335-356, doi:10.1016/0167-2789(89)90074-2. · Zbl 0671.62099 |
| [16] | Coifman, R. and Lafon, S., Diffusion maps, Appl. Comput. Harmon. Anal., 21 (2006), pp. 5-30, doi:10.1016/j.acha.2006.04.006. · Zbl 1095.68094 |
| [17] | Collet, P. and Eckmann, J.-P., Liapunov multipliers and decay of correlations in dynamical systems, J. Stat. Phys., 115 (2004), pp. 217-254, doi:10.1023/B:JOSS.0000019817.71073.61. · Zbl 1157.37306 |
| [18] | Danforth, C. and Yorke, J., Making forecasts for chaotic physical processes, Phys. Rev. Lett., 96 (2006), 144102, doi:10.1103/PhysRevLett.96.144102. |
| [19] | Das, S. and Giannakis, D., Delay-coordinate maps and the spectra of Koopman operators, J. Stat. Phys., 175 (2019), pp. 1107-1145, doi:10.1007/s10955-019-02272-w. · Zbl 1459.37023 |
| [20] | Das, S. and Giannakis, D., Koopman spectra in reproducing kernel Hilbert spaces, Appl. Comput. Harmon. Anal., 49 (2020), pp. 573-607, doi:10.1016/j.acha.2020.05.008. · Zbl 1446.37072 |
| [21] | Das, S. and Giannakis, D., Reproducing kernel Hilbert algebras on compact Lie groups, J. Funct. Anal. Appl., 29 (2023), doi:10.1007/s00041-023-09992-4. |
| [22] | Das, S., Mustavee, S., and Agarwal, S., Uncovering Quasi-Periodic Nature of Physical Systems: A Case Study of Signalized Intersections, https://arxiv.org/abs/2109.08623, 2021. |
| [23] | Das, S. and Yorke, J., Super convergence of ergodic averages for quasiperiodic orbits, Nonlinearity, 31 (2018), pp. 491-501, doi:10.1088/1361-6544/aa99a0. · Zbl 1384.37010 |
| [24] | Dechert, W. and Gençay, R., The topological invariance of Lyapunov exponents in embedded dynamics, Phys. D, 90 (1996), pp. 40-55, doi:10.1016/0167-2789(95)00225-1. · Zbl 0884.34059 |
| [25] | Dechert, W. and Gençay, R., Is the largest Lyapunov exponent preserved in embedded dynamics?, Phys. Lett. A, 276 (2000), pp. 59-64, doi:10.1016/S0375-9601(00)00657-5. · Zbl 1119.37305 |
| [26] | Duarte, P. and Klein, S., Lyapunov Exponents of Linear Cocycles, , Atlantis Press, 2016, doi:10.2991/978-94-6239-124-6. · Zbl 1364.37001 |
| [27] | Froyland, G., Lloyd, S., and Quas, A., Coherent structures and isolated spectrum for Perron-Frobenius cocycles, Ergodic Theory Dynam. Systems, 30 (2010), pp. 729-756, doi:10.1017/S0143385709000339. · Zbl 1205.37015 |
| [28] | Furstenberg, H. and Kesten, H., Products of random matrices, Ann. Math. Stat., 31 (1960), pp. 457-469, doi:10.1214/aoms/1177705909. · Zbl 0137.35501 |
| [29] | Furstenberg, H. and Kifer, Y., Random matrix products and measures on projective spaces, Israel J. Math., 46 (1983), pp. 12-32, doi:10.1007/BF02760620. · Zbl 0528.60028 |
| [30] | Gallant, Nonlinear regression, Amer. Statist., 29 (1975), pp. 73-81, doi:10.1080/00031305.1975.10477374. · Zbl 0328.62043 |
| [31] | Gauthier, D., Bollt, E., Griffith, A., and Barbosa, W., Next generation reservoir computing, Nature Comm., 12 (2021), pp. 1-8, doi:10.1038/s41467-021-25801-2. |
| [32] | Giannakis, D., Das, S., and Slawinska, J., Reproducing kernel Hilbert space compactification of unitary evolution groups, Appl. Comput. Harmon. Anal., 54 (2021), pp. 75-136, doi:10.1016/j.acha.2021.02.004. · Zbl 1473.37101 |
| [33] | Giannakis, D., Slawinska, J., and Zhao, Z., Spatiotemporal feature extraction with data-driven Koopman operators, J. Mach. Learn. Res. Proc., 44 (2015), pp. 103-115. |
| [34] | Gonon, L. and Ortega, J., Fading memory echo state networks are universal, Neural Networks, 138 (2021), pp. 10-13, doi:10.1016/j.neunet.2021.01.025. · Zbl 1521.68119 |
| [35] | Gouëzel, S. and Stoyanov, L., Quantitative Pesin theory for Anosov diffeomorphisms and flows, Ergodic Theory Dynam. Systems, 39 (2019), pp. 159-200, doi:10.1017/etds.2017.25. · Zbl 1404.37062 |
| [36] | Grigoryeva, L., Hart, A., and Ortega, J., Chaos on compact manifolds: Differentiable synchronizations beyond the Takens theorem, Phys. Rev. E, 103 (2021), 062204, doi:10.1103/PhysRevE.103.062204. |
| [37] | Grigoryeva, L., Hart, A., and Ortega, J., Learning Strange Attractors with Reservoir Systems, https://arxiv.org/abs/2108.05024, 2021. · Zbl 1525.37020 |
| [38] | Grigoryeva, L., Henriques, J., Larger, L., and Ortega, J., Stochastic nonlinear time series forecasting using time-delay reservoir computers: Performance and universality, Neural Networks, 55 (2014), pp. 59-71, doi:10.1016/j.neunet.2014.03.004. · Zbl 1308.68095 |
| [39] | Halmos, P., Lectures on Ergodic Theory, American Mathematical Society, 1956. · Zbl 0073.09302 |
| [40] | Hamilton, F., Berry, T., and Sauer, T., Ensemble Kalman filtering without a model, Phys. Rev. X, 6 (2016), 011021, doi:10.1103/PhysRevX.6.011021. |
| [41] | Hamilton, F., Berry, T., and Sauer, T., Kalman-Takens filtering in the presence of dynamical noise, Eur. Phys. J. Special Topics, 226 (2017), pp. 3239-3250, doi:10.1140/epjst/e2016‐60363-2. |
| [42] | Hammel, S., Yorke, J., and Grebogi, C., Numerical orbits of chaotic processes represent true orbits, Bull. Amer. Math. Soc., 19 (1988), pp. 465-469, doi:10.1090/S0273-0979-1988‐15701-1. · Zbl 0689.58026 |
| [43] | Harlim, J., Jiang, S., Liang, S., and Yang, H., Machine learning for prediction with missing dynamics, J. Comput. Phys., 428 (2021), 109922, doi:10.1016/j.jcp.2020.109922. · Zbl 07511409 |
| [44] | Hart, A., Hook, J., and Dawes, J., Embedding and approximation theorems for echo state networks, Neural Networks, 128 (2020), pp. 234-247, doi:10.1016/j.neunet.2020.05.013. · Zbl 1468.68098 |
| [45] | Hart, A., Hook, J., and Dawes, J., Echo state networks trained by Tikhonov least squares are \(L^2(\mu)\) approximators of ergodic dynamical systems, Phys. D, 421 (2021), 132882, doi:10.1016/j.physd.2021.132882. · Zbl 1491.37075 |
| [46] | Hochreiter, S. and Schmidhuber, J., Long short-term memory, Neural Comput., 9 (1997), pp. 1735-1780, doi:10.1162/neco.1997.9.8.1735. |
| [47] | Jaeger, H. and Haas, H., Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication, Science, 304 (2004), pp. 78-80, doi:10.1126/science.1091277. |
| [48] | Jimenez, K., Moreno, J., and Ruggeri, G., Forecasting on chaotic time series: A local optimal linear-reconstruction method, Phys. Rev. A, 45 (1992), pp. 3553-3558, doi:10.1103/PhysRevA.45.3553. |
| [49] | Katok, A., Lyapunov exponents, entropy and periodic orbits for diffeomorphisms, Publ. Math. Inst. Hautes Études Sci., 51 (1980), pp. 137-173, doi:10.1007/BF02684777. · Zbl 0445.58015 |
| [50] | Korda, M. and Mezić, I., Optimal construction of Koopman eigenfunctions for prediction and control, IEEE Trans. Automat. Control, 65 (2020), pp. 5114-5129, doi:10.1109/TAC.2020.2978039. · Zbl 1536.93309 |
| [51] | Kugiumtzis, D., Lingjaerde, O. C., and Christophersen, N., Regularized local linear prediction of chaotic time series, Phys. D, 112 (1998), pp. 344-360, doi:10.1016/s0167-2789(97)00171-1. · Zbl 0931.37037 |
| [52] | LeCun, Y., Bengio, Y., and Hinton, G., Deep learning, Nature, 521 (2015), pp. 436-444, doi:10.1038/nature14539. |
| [53] | Ledrappier, F. and Young, L.-S., The metric entropy of diffeomorphisms: Part I: Characterization of measures satisfying Pesin’s entropy formula, Ann. of Math. (2), (1985), pp. 509-539, doi:10.2307/1971328. · Zbl 0605.58028 |
| [54] | Ledrappier, F. and Young, L.-S., The metric entropy of diffeomorphisms: Part II: Relations between entropy, exponents and dimension, Ann. of Math. (2), (1985), pp. 540-574, doi:10.2307/1971329. · Zbl 1371.37012 |
| [55] | Liao, G., Sun, W., Vargas, E. and Wang, S., Approximation of Bernoulli measures for non-uniformly hyperbolic systems, Ergodic Theory Dynam. Systems, 40 (2022), pp. 233-247, doi:10.1017/etds.2018.33. · Zbl 1427.37024 |
| [56] | Logunov, M. and Butkovskii, O., Mixing and Lyapunov exponents of chaotic systems, Tech. Phys., 53 (2008), pp. 959-965, doi:10.1134/S106378420808001X. |
| [57] | Lu, Z., Pathak, J., Hunt, B., Girvan, M., Brockett, R., and Ott, E., Reservoir observers: Model-free inference of unmeasured variables in chaotic systems, Chaos, 27 (2017), p. 041102, doi:10.1063/1.4979665. |
| [58] | Lukoševičius, M. and Jaeger, H., Reservoir computing approaches to recurrent neural network training, Comput. Sci. Rev., 3 (2009), pp. 127-149, doi:10.1016/j.cosrev.2009.03.005. · Zbl 1302.68235 |
| [59] | Ma, C., Wang, J., and W. E, Model reduction with memory and the machine learning of dynamical systems, Commun. Comput. Phys., 25 (2018), pp. 947-962, doi:10.4208/CICP.OA-2018-0269. · Zbl 1473.35450 |
| [60] | Matthews, M., On the Uniform Approximation of Nonlinear Discrete-Time Fading-Memory Systems Using Neural Network Models, Ph.D. thesis, ETH Zurich, 1992. |
| [61] | Maulik, R., A. Mohan, Lusch, B., Madireddy, S., Balaprakash, P., and Livescu, D., Time-series learning of latent-space dynamics for reduced-order model closure, Phys. D, 405 (2020), 132368, doi:10.1016/j.physd.2020.132368. · Zbl 07477434 |
| [62] | Mustavee, S., Agarwal, S., Enyioha, C., and Das, S., A linear dynamical perspective on epidemiology: Interplay between early Covid-19 outbreak and human activity, Nonlinear Dyn., 109 (2022), pp. 1233-1252, doi:10.1007/s11071-022-07469-5. |
| [63] | Pathak, J., Lu, Z., Hunt, R. B., Girvan, M., and Ott, E., Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data, Chaos, 27 (2017), 121102, doi:10.1063/1.5010300. · Zbl 1390.37138 |
| [64] | Pesin, Y., Families of invariant manifolds corresponding to nonzero characteristic exponents, Math. USSR-Izvestiya, 10 (1976), p. 1261, doi:10.1070/IM1976v010n06ABEH001835. · Zbl 0383.58012 |
| [65] | Rahman, R. and Hasan, S., Real-time signal queue length prediction using long short-term memory neural network, Neural Comp. Appl., 33 (2021), pp. 3311-3324, doi:10.1007/s00521-020‐05196-9. |
| [66] | Ruelle, D., Ergodic theory of differentiable dynamical systems, Publ. Math. Inst. Hautes Études Sci., 50 (1979), pp. 27-58, doi:10.1007/BF02684768. · Zbl 0426.58014 |
| [67] | Sauer, T., Tempkin, J., and Yorke, J., Spurious Lyapunov exponents in attractor reconstruction, Phys. Rev. Lett, 81 (1998), pp. 4341-4344, doi:10.1103/PhysRevLett.81.4341. |
| [68] | Sauer, T., Yorke, J. A., and Casdagli, M., Embedology, J. Stat. Phys., 65 (1991), pp. 579-616, doi:10.1007/bf01053745. · Zbl 0943.37506 |
| [69] | Skufca, J. and Bollt, E., Relaxing conjugacy to fit modeling in dynamical systems, Phys. Rev. E, 76 (2007), 026220, doi:10.1103/PhysRevE.76.026220. |
| [70] | Skufca, J. and Bollt, E., A concept of homeomorphic defect for defining mostly conjugate dynamical systems, Chaos, 18 (2008), 013118, doi:10.1063/1.2837397. · Zbl 1306.37021 |
| [71] | Slawinska, J. and Giannakis, D., Indo-Pacific variability on seasonal to multidecadal time scales. Part I: Intrinsic SST modes in models and observations, J. Climate, 30 (2017), pp. 5265-5294, doi:10.1175/JCLI-D-16-0176.1. |
| [72] | Smith, L., Identification and prediction of low dimensional dynamics, Phys. D, 58 (1992), pp. 50-76, doi:10.1016/0167-2789(92)90101-R. · Zbl 1194.37142 |
| [73] | Stark, J., Regularity of invariant graphs for forced systems, Ergodic Theory Dynam. Systems, 19 (1999), pp. 155-199, doi:10.1017/S0143385799126555. · Zbl 1004.37018 |
| [74] | Sugihara, G. and May, R. M., Nonlinear forecasting as a way of distinguishing chaos from measurement error in time series, Nature, 344 (1990), pp. 734-741, doi:10.1038/344734a0. |
| [75] | Trillos, N. and Slepčev, D., A variational approach to the consistency of spectral clustering, Appl. Comput. Harmon. Anal., 45 (2018), pp. 239-281, doi:10.1016/j.acha.2016.09.003. · Zbl 1396.49013 |
| [76] | Viana, M., (Dis)continuity of lyapunov exponents, Ergodic Theory Dynam. Systems, 40 (2020), pp. 577-611, doi:10.1017/etds.2018.50. · Zbl 1442.37048 |
| [77] | Cowieson, L. Y. W., SRB measures as zero-noise limits, Ergodic Theory Dynam. Systems, 25 (2005), pp. 1115-1138, doi:10.1017/S0143385704000604. · Zbl 1098.37020 |
| [78] | Wang, Z. and Sun, W., Lyapunov exponents of hyperbolic measures and hyperbolic periodic orbits, Trans. Amer. Math. Soc., 362 (2010), pp. 4267-4282, https://www.jstor.org/stable/25733367. · Zbl 1201.37025 |
| [79] | Young, L., Statistical properties of dynamical systems with some hyperbolicity, Ann. of Math. (2), 147 (1998), pp. 585-650, doi:10.2307/120960. · Zbl 0945.37009 |
| [80] | Young, L., Recurrence times and rates of mixing, Israel J. Math., 110 (1999), pp. 153-188, doi:10.1007/BF02808180. · Zbl 0983.37005 |
| [81] | Zhao, Z. and Giannakis, D., Analog forecasting with dynamics-adapted kernels, Nonlinearity, 29 (2016), pp. 2888-2939, doi:10.1088/0951-7715/29/9/2888. · Zbl 1365.37062 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
