Neural network-based parametric system identification: a review. (English) Zbl 1533.93109
Parametric system identification methods using neural networks are analysed. Namely, authors focused on the feedforward neural networks, recurrent neural networks and encoder-decoder. It was outlined that the rigorous mathematical interpretation of neural networks is an essential stage in parametric system identification. From the literature overview it was concluded that no results using neural networks available for the nonstationary dynamic systems identification and modelling with coloured noise.
Reviewer: Denis Sidorov (Irkutsk)
MSC:
| 93B30 | System identification |
| 93C10 | Nonlinear systems in control theory |
| 68T07 | Artificial neural networks and deep learning |
Keywords:
nonstationary time series; nonlinear system identification; Granger causality; robust and adaptive modelling; interpretable deep learningReferences:
| [1] | Acuna, G.; Ramirez, C.; Curilem, M., Comparing NARX and NARMAX models using ANN and SVM for cash demand forecasting for ATM (2012) · doi:10.1109/IJCNN.2012.6252476 |
| [2] | Aggarwal, C. C., Neural networks and deep learning, Springer, 10, 978, 3 (2018) · Zbl 1402.68001 · doi:10.1007/978-3-319-94463-0 |
| [3] | Ahmed, N. K.; Atiya, A. F.; el Gayar, N.; El-Shishiny, H., An empirical comparison of machine learning models for time series forecasting, Econometric Reviews, 29, 5, 594-621 (2010) · doi:10.1080/07474938.2010.481556 |
| [4] | Ardalani-Farsa, M.; Zolfaghari, S., Chaotic time series prediction with residual analysis method using hybrid elman-NARX neural networks, Neurocomputing, 73, 13-15, 2540-2553 (2010) · doi:10.1016/j.neucom.2010.06.004 |
| [5] | Bahdanau, D., Cho, K. H., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. 3rd International conference on learning representations, ICLR 2015 - Conference Track Proceedings. https://arxiv.org/abs/1409.0473v7. |
| [6] | Bai, S., Kolter, J. Z., & Koltun, V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. http://arxiv.org/abs/1803.01271. |
| [7] | Billings, S. A., Nonlinear system identification: NARMAX methods in the time, frequency, and spatio-temporal domains, Introduction, 1-14 (2013), John Wiley & Sons, Ltd · Zbl 1287.93101 · doi:10.1002/9781118535561 |
| [8] | Billings, S. A.; Wei, H.-L.; Balikhin, M. A., Generalized multiscale radial basis function networks, Neural Networks, 20, 10, 1081-1094 (2007) · Zbl 1254.68197 · doi:10.1016/j.neunet.2007.09.017 |
| [9] | Bodhanwala, M. (2014). A novel non-linear GARCH framework for modelling the volatility of heteroskedastic time series. PhD Thesis, University of Sheffield. |
| [10] | Box, G. E. P., Jenkins, G. M., Reinsel, G. C., & Ljung, G. M. (2015). Time series analysis : Forecasting and control (5th ed.). · Zbl 1317.62001 |
| [11] | Brunton, S. L.; Proctor, J. L.; Kutz, J. N., Discovering governing equations from data by sparse identification of nonlinear dynamical systems, Proceedings of the National Academy of Sciences, 113, 15, 3932-3937 (2016) · Zbl 1355.94013 · doi:10.1073/pnas.1517384113 |
| [12] | Cao, Y.; Ding, Y.; Jia, M.; Tian, R., A novel temporal convolutional network with residual self-attention mechanism for remaining useful life prediction of rolling bearings, Reliability Engineering and System Safety, 215 (2021) · doi:10.1016/J.RESS.2021.107813 |
| [13] | Chen, H.; Gong, Y.; Hong, X.; Chen, S., A fast adaptive tunable RBF network for nonstationary systems, IEEE Transactions on Cybernetics, 46, 12, 2683-2692 (2016) · doi:10.1109/TCYB.2015.2484378 |
| [14] | Chen, S., Nonlinear time series modelling and prediction using Gaussian RBF networks with enhanced clustering and RLS learning, Electronics Letters, 31, 2, 117-118 (1995) · doi:10.1049/el:19950085 |
| [15] | Chng, E. S.; Chen, S.; Mulgrew, B., Gradient radial basis function networks for nonlinear and nonstationary time series prediction, IEEE Transactions on Neural Networks, 7, 1, 190-194 (1996) · doi:10.1109/72.478403 |
| [16] | Cui, P.; Athey, S., Stable learning establishes some common ground between causal inference and machine learning, Nature Machine Intelligence, 4, 2, 110-115 (2022) · doi:10.1038/s42256-022-00445-z |
| [17] | Ditzler, G.; Roveri, M.; Alippi, C.; Polikar, R., Learning in nonstationary environments: A survey, IEEE Computational Intelligence Magazine, 10, 4, 12-25 (2015) · doi:10.1109/MCI.2015.2471196 |
| [18] | Du, Y., Wang, J., Feng, W., Pan, S., Qin, T., Xu, R., & Wang, C. (2021). AdaRNN: Adaptive learning and forecasting of time series. Proceedings of the 30th ACM International conference on information & knowledge management, 402-411. . |
| [19] | Elman, J. L., Finding structure in time, Cognitive Science, 14, 2, 179-211 (1990) · doi:10.1207/s15516709cog1402_1 |
| [20] | Fan, J.; Zhang, K.; Huang, Y.; Zhu, Y.; Chen, B., Parallel spatio-temporal attention-based TCN for multivariate time series prediction, Neural Computing and Applications, 1-10 (2021) · doi:10.1007/S00521-021-05958-Z/FIGURES/7 |
| [21] | Fung, C. F.; Billings, S. A.; Zhang, H., Generalised transfer functions of neural networks, Mechanical Systems and Signal Processing, 11, 6, 843-868 (1997) · doi:10.1006/mssp.1997.0112 |
| [22] | Gauthier, D. J.; Bollt, E.; Griffith, A.; Barbosa, W. A. S., Next generation reservoir computing, Nature Communications, 12, 1, 1-8 (2021) · doi:10.1038/s41467-021-25801-2 |
| [23] | Goodfellow, I.; Bengio, Y.; Courville, A., Deep learning (2016), MIT Press · Zbl 1373.68009 |
| [24] | Gu, Y.; Wei, H. L.; Boynton, R. J.; Walker, S. N.; Balikhin, M. A., System identification and data-driven forecasting of AE index and prediction uncertainty analysis using a new cloud-NARX model, Journal of Geophysical Research: Space Physics, 124, 1, 248-263 (2019) · doi:10.1029/2018JA025957 |
| [25] | Hao, H., Wang, Y., Xia, Y., Zhao, J., & Shen, F. (2020). Temporal convolutional attention-based network for sequence modeling. https://arxiv.org/abs/2002.12530v2. |
| [26] | He, F.; Billings, S. A.; Wei, H. L.; Sarrigiannis, P. G.; Zhao, Y., Spectral analysis for nonstationary and nonlinear systems: A discrete-time-model-based approach, IEEE Transactions on Bio-Medical Engineering, 60, 8, 2233-2241 (2013) · doi:10.1109/TBME.2013.2252347 |
| [27] | He, F.; Wei, H.-L.; Billings, S. A., Identification and frequency domain analysis of non-stationary and nonlinear systems using time-varying NARMAX models, International Journal of Systems Science, 46, 11, 2087-2100 (2015) · Zbl 1332.93356 · doi:10.1080/00207721.2013.860202 |
| [28] | He, F.; Yang, Y., Nonlinear system identification of neural systems from neurophysiological signals, Neuroscience, 458, 213-228 (2021) · doi:10.1016/j.neuroscience.2020.12.001 |
| [29] | Hong, X.; Mitchell, R. J.; Chen, S.; Harris, C. J.; Li, K.; Irwin, G. W., Model selection approaches for non-linear system identification: A review, International Journal of Systems Science, 39, 10, 925-946 (2008) · Zbl 1233.93097 · doi:10.1080/00207720802083018 |
| [30] | Hornik, K.; Stinchcombe, M.; White, H., Multilayer feedforward networks are universal approximators, Neural Networks, 2, 5, 359-366 (1989) · Zbl 1383.92015 · doi:10.1016/0893-6080(89)90020-8 |
| [31] | Jiao, H., Zhang, L., Shen, Q., Zhu, J., & Shi, P. (2018). Robust gene circuit control design for time-delayed genetic regulatory networks without SUM regulatory logic. . |
| [32] | Jumilla-Corral, A. A.; Perez-Tello, C.; Campbell-Ramírez, H. E.; Medrano-Hurtado, Z. Y.; Mayorga-Ortiz, P.; Avitia, R. L., Modeling of harmonic current in electrical grids with photovoltaic power integration using a nonlinear autoregressive with external input neural networks, Energies, 14, 13, 4015 (2021) · doi:10.3390/EN14134015 |
| [33] | Kaiser, E.; Kutz, J. N.; Brunton, S. L., Sparse identification of nonlinear dynamics for model predictive control in the low-data limit, Proceedings of the Royal Society A, 474, 2219 (2018) · Zbl 1425.93175 · doi:10.1098/RSPA.2018.0335 |
| [34] | Khanna, S., & Tan, V. Y. F. (2019). Economy statistical recurrent units for inferring nonlinear granger causality. https://arxiv.org/abs/1911.09879v2. |
| [35] | Kitaev, N., Kaiser, Ł, & Levskaya, A. (2020). Reformer: The efficient transformer. ICLR. http://arxiv.org/abs/2001.04451. |
| [36] | Li, L. M.; Billings, S. A., Discrete time subharmonic modelling and analysis, International Journal of Control, 78, 16, 1265-1284 (2005) · Zbl 1090.93003 · doi:10.1080/00207170500293594 |
| [37] | Li, T.; Li, G.; Xue, T.; Zhang, J., Analyzing brain connectivity in the mutual regulation of emotion – movement using bidirectional granger causality, Frontiers in Neuroscience, 14, 369 (2020) · doi:10.3389/fnins.2020.00369 |
| [38] | Li, Y.; Cui, W.-G.; Guo, Y.-Z.; Huang, T.; Yang, X.-F.; Wei, H.-L., Time-varying system identification using an ultra-orthogonal forward regression and multiwavelet basis functions with applications to Eeg, IEEE Transactions on Neural Networks and Learning Systems, 1-13 (2017) · doi:10.1109/TNNLS.2017.2709910 |
| [39] | Li, Y.; Wei, H. L.; Billings, S. A.; Sarrigiannis, P. G., Identification of nonlinear time-varying systems using an online sliding-window and common model structure selection (CMSS) approach with applications to EEG, International Journal of Systems Science, 47, 11, 2671-2681 (2016) · Zbl 1345.93160 · doi:10.1080/00207721.2015.1014448 |
| [40] | Lim, B.; Zohren, S., Time-series forecasting with deep learning: A survey, Philosophical Transactions of the Royal Society A, 379, 2194, 20200209 (2021) · doi:10.1098/rsta.2020.0209 |
| [41] | Liu, B.; He, X.; Song, M.; Li, J.; Qu, G.; Lang, J.; Gu, R., A method for mining granger causality relationship on atmospheric visibility, ACM Transactions on Knowledge Discovery from Data (TKDD), 15, 5 (2021) · doi:10.1145/3447681 |
| [42] | Liu, T.; Chen, S.; Liang, S.; Du, D.; Harris, C. J., Fast tunable gradient RBF networks for online modeling of nonlinear and nonstationary dynamic processes, Journal of Process Control, 93, 53-65 (2020) · doi:10.1016/j.jprocont.2020.07.009 |
| [43] | Liu, T.; Chen, S.; Liang, S.; Gan, S.; Harris, C. J., Fast adaptive gradient RBF networks for online learning of nonstationary time series, IEEE Transactions on Signal Processing, 68, 2015-2030 (2020) · Zbl 07590874 · doi:10.1109/TSP.2020.2981197 |
| [44] | Ljung, L.; Andersson, C.; Tiels, K.; Schön, T. B., Deep learning and system identification, IFAC-PapersOnLine, 53, 2, 1175-1181 (2020) · doi:10.1016/j.ifacol.2020.12.1329 |
| [45] | Montalto, A.; Stramaglia, S.; Faes, L.; Tessitore, G.; Prevete, R.; Marinazzo, D., Neural networks with non-uniform embedding and explicit validation phase to assess Granger causality, Neural Networks, 71, 159-171 (2015) · doi:10.1016/j.neunet.2015.08.003 |
| [46] | Moody, J., & Darken, C. (1988). Learning with localized receptive fields. |
| [47] | Nauta, M.; Bucur, D.; Seifert, C., Causal discovery with attention-based convolutional neural networks, Machine Learning and Knowledge Extraction, 1, 1, 312-340 (2019) · doi:10.3390/MAKE1010019 |
| [48] | Neshat, N.; Hadian, H.; Behzad, M., Nonlinear ARIMAX model for long – term sectoral demand forecasting, Management Science Letters, 8, 6, 581-592 (2018) · doi:10.5267/j.msl.2018.4.032 |
| [49] | Oord, A. van den, Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016). WaveNet: A generative model for raw audio. ArXiv Preprint, ArXiv:1609.03499. http://arxiv.org/abs/1609.03499. |
| [50] | Oreshkin, B. N., Carpov, D., Chapados, N., & Bengio, Y. (2020, May 24). N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. ICLR. http://arxiv.org/abs/1905.10437. |
| [51] | Pérez-Sánchez, B.; Fontenla-Romero, O.; Guijarro-Berdiñas, B., A review of adaptive online learning for artificial neural networks, Artificial Intelligence Review, 49, 2, 281-299 (2018) · doi:10.1007/s10462-016-9526-2 |
| [52] | Rosoł, M.; Młyńczak, M.; Cybulski, G., Granger causality test with nonlinear neural-network-based methods: Python package and simulation study, Computer Methods and Programs in Biomedicine, 106669 (2022) · doi:10.1016/J.CMPB.2022.106669 |
| [53] | Rudin, C., Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, Nature Machine Intelligence, 1, 5, 206-215 (2019) · doi:10.1038/s42256-019-0048-x |
| [54] | Schön, T. B.; Lindsten, F.; Dahlin, J.; Wågberg, J.; Naesseth, C. A.; Svensson, A.; Dai, L., Sequential Monte Carlo methods for system identification, IFAC-PapersOnLine, 48, 28, 775-786 (2015) · doi:10.1016/j.ifacol.2015.12.224 |
| [55] | Sen, R.; Yu, H. F.; Dhillon, I., Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting, Advances in Neural Information Processing Systems, 32 (2019) · doi:10.48550/arxiv.1905.03806 |
| [56] | Shen, Q.; Shi, P.; Agarwal, R. K.; Shi, Y., Adaptive neural network-based filter design for nonlinear systems with multiple constraints, IEEE Transactions on Neural Networks and Learning Systems, 32, 7, 3256-3261 (2021) · doi:10.1109/TNNLS.2020.3009391 |
| [57] | Shen, Q.; Shi, P.; Zhu, J.; Wang, S.; Shi, Y., Neural networks-based distributed adaptive control of nonlinear multiagent systems, IEEE Transactions on Neural Networks and Learning Systems, 31, 3, 1010-1021 (2020) · doi:10.1109/TNNLS.2019.2915376 |
| [58] | Tang, P.; Du, P.; Xia, J.; Zhang, P.; Zhang, W., Channel attention-based temporal convolutional network for satellite image time series classification, IEEE Geoscience and Remote Sensing Letters, 19 (2022) · doi:10.1109/LGRS.2021.3095505 |
| [59] | Tank, A.; Covert, I.; Foti, N.; Shojaie, A.; Fox, E. B., Neural granger causality, IEEE Transactions on Pattern Analysis and Machine Intelligence, 1-1 (2021) · doi:10.1109/TPAMI.2021.3065601 |
| [60] | Tutunji, T. A., Parametric system identification using neural networks, Applied Soft Computing, 47, 251-261 (2016) · doi:10.1016/j.asoc.2016.05.012 |
| [61] | Vaswani, A.; Brain, G.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; Polosukhin, I., Attention is all you need, Advances in Neural Information Processing Systems, 30 (2017) · doi:10.48550/arXiv.1706.03762 |
| [62] | Wang, H.; Song, G., Innovative NARX recurrent neural network model for ultra-thin shape memory alloy wire, Neurocomputing, 134, 289-295 (2014) · doi:10.1016/j.neucom.2013.09.050 |
| [63] | Wang, X.; Han, M., Online sequential extreme learning machine with kernels for nonstationary time series prediction, Neurocomputing, 145, 90-97 (2014) · doi:10.1016/j.neucom.2014.05.068 |
| [64] | Wang, Y.; Lin, K.; Qi, Y.; Lian, Q.; Feng, S.; Wu, Z.; Pan, G., Estimating brain connectivity with varying-length time lags using a recurrent neural network, IEEE Transactions on Biomedical Engineering, 65, 9, 1953-1963 (2018) · doi:10.1109/TBME.2018.2842769 |
| [65] | Wei, H. L.; Billings, S. A., Long term prediction of non-linear time series using multiresolution wavelet models, International Journal of Control, 79, 6, 569-580 (2006) · Zbl 1179.37111 · doi:10.1080/00207170600621447 |
| [66] | Wei, H. L.; Billings, S. A., Feature subset selection and ranking for data dimensionality reduction, IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 1, 162-166 (2007) · doi:10.1109/TPAMI.2007.250607 |
| [67] | Werbos, P. J., Backpropagation through time: What it does and how to do it, Proceedings of the IEEE, 78, 10, 1550-1560 (1990) · doi:10.1109/5.58337 |
| [68] | Wu, H., Xu, J., Wang, J., & Long, M. (2021). Autoformer: Decomposition transformers with auto-correlation for long-Term series forecasting. 35th Conference on Neural Information Processing Systems (NeurIPS). http://arxiv.org/abs/2106.13008. |
| [69] | Zhao, Y.; Bigg, G. R.; Billings, S. A.; Hanna, E.; Sole, A. J.; Wei, H. L.; Kadirkamanathan, V.; Wilton, D. J., Inferring the variation of climatic and glaciological contributions to West Greenland iceberg discharge in the twentieth century, Cold Regions Science and Technology, 121, 167-178 (2016) · doi:10.1016/j.coldregions.2015.08.006 |
| [70] | Zhao, Y.; Billings, S. A.; Wei, H.; He, F.; Sarrigiannis, P. G., A new NARX-based Granger linear and nonlinear casual influence detection method with applications to EEG data, Journal of Neuroscience Methods, 212, 1, 79-86 (2013) · doi:10.1016/j.jneumeth.2012.09.019 |
| [71] | Zhao, Y.; Billings, S. A.; Wei, H.; Sarrigiannis, P. G., Tracking time-varying causality and directionality of information flow using an error reduction ratio test with applications to electroencephalography data, Physical Review E, 86, 5, 051919 (2012) · doi:10.1103/PhysRevE.86.051919 |
| [72] | Zhao, Y.; Hanna, E.; Bigg, G. R.; Zhao, Y., Tracking nonlinear correlation for complex dynamic systems using a windowed error reduction ratio method, Complexity, 2017 (2017) · Zbl 1377.93026 · doi:10.1155/2017/8570720 |
| [73] | Zhou, H.; Zhang, S.; Peng, J.; Zhang, S.; Li, J.; Xiong, H.; Zhang, W., Informer: Beyond efficient transformer for long sequence time-series forecasting, Proceedings of the AAAI Conference on Artificial Intelligence, 35, 12, 11106-11115 (2021) |
| [74] | Zhu, Y.-P.; Yuan, J.; Lang, Z. Q.; Schwingshackl, C. W.; Salles, L.; Kadirkamanathan, V., The data-driven surrogate model-based dynamic design of aeroengine fan systems, Journal of Engineering for Gas Turbines and Power, 143, 10 (2021) · doi:10.1115/1.4049504 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
