A novel recurrent neural network of gated unit based on Euler’s method and application. (English) Zbl 07918192
Summary: The main focus of this paper is to interpret the neural networks as the discretizations of differential equations, which has more benefits for analyzing the intrinsic mechanisms of neural networks. Under this theoretical framework, we propose a conditionally stable network unit called the GUEM, which is based on the Euler’s method for ordinary differential equations and the gated thought in recurrent neural networks. Moreover, we build a sequence-to-sequence recurrent neural network based on the GUEM and fully connected layers, which does not amplify perturbations caused by noises of the input features. Finally, the numerical experiments of inverse scattering problems demonstrate the effectiveness and efficiency of our network.
MSC:
| 34D20 | Stability of solutions to ordinary differential equations |
| 65L07 | Numerical investigation of stability of solutions to ordinary differential equations |
| 35R30 | Inverse problems for PDEs |
| 68T07 | Artificial neural networks and deep learning |
Keywords:
ordinary differential equation; recurrent neural network; stability; inverse scattering problemReferences:
| [1] | A. A. Adler Boyle, Lectrical impedance tomography: Tissue properties to image measures, IEEE Transactions on Biomedical Engineering, 64, 2494-2504 (2017) |
| [2] | Y. Bengio, Learning deep architectures for AI, Foundations and Trends in Machine Learning, 2, 1-127 (2009) · Zbl 1192.68503 |
| [3] | X. H. H. Cao Diao Liu, On nodal and singular structures of Laplacian eigenfunctions and applications to inverse scattering problems, Journal de Mathematiques Pures et Appliquees, 143, 116-161 (2020) · Zbl 1452.35120 · doi:10.1016/j.matpur.2020.09.011 |
| [4] | X. H. H. Cao Diao Liu, On novel geometric structures of Laplacian eigenfunctions in \(\mathbb{R}^3\) and applications to inverse problems, SIAM Journal on Mathematical Analysis, 53, 1263-1294 (2021) · Zbl 1461.35167 · doi:10.1137/19M1292989 |
| [5] | B. Chang, M. Chen, E. Haber, et al., AntisymmetricRNN: A dynamical system view on recurrent neural networks, Proceedings of the International Conference on Learning Representations, New Orleans, USA, (2019), 1-15. |
| [6] | B. Chang, L. Meng, E. Haber, et al., Reversible architectures for arbitrarily deep residual neural networks, Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, USA, (2018), 2811-2818. |
| [7] | R. Chen, Y. Rubanova, J. Bettencourt, et al., Neural ordinary differential equations, Proceedings of the International Conference on Neural Information Processing Systems, Montreal, Canada, (2018), 6572-6583. |
| [8] | K. Cho, B. Merri \(\ddot{e}\) nboer, C. Gulcehre, et al., Learning phrase representations using RNN encoder-decoder for statistical machine translation, Proceedings of the Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, (2014), 1724-1734. |
| [9] | D. J. P. Colton Coyle Monk, Recent developments in inverse acoustic scattering theory, Siam Review, 42, 369-414 (2000) · Zbl 0960.76081 · doi:10.1137/S0036144500367337 |
| [10] | G. Cybenko, Approximation by superpositions of a sigmoidal function, Mathematics of Control, Signals and Systems, 2, 303-314 (1989) · Zbl 0679.94019 · doi:10.1007/BF02551274 |
| [11] | Y. J. H. Deng Li Liu, On identifying magnetized anomalies using geomagnetic monitoring, Archive for Rational Mechanics and Analysis, 231, 153-187 (2019) · Zbl 1405.78002 · doi:10.1007/s00205-018-1276-7 |
| [12] | Y. J. H. Deng Li Liu, On identifying magnetized anomalies using geomagnetic monitoring within a magnetohydrodynamic model, Archive for Rational Mechanics and Analysis, 235, 691-721 (2020) · Zbl 1434.35243 · doi:10.1007/s00205-019-01429-x |
| [13] | Y. H. W. Deng Liu Tsui, Identifying variations of magnetic anomalies using geomagnetic monitoring, Discrete and Continuous Dynamical Systems, 40, 6411-6440 (2020) · Zbl 1448.35494 · doi:10.3934/dcds.2020285 |
| [14] | Y. H. G. Deng Liu Uhlmann, On an inverse boundary problem arising in brain imaging, Journal of Differential Equations, 267, 2471-2502 (2019) · Zbl 1416.35254 · doi:10.1016/j.jde.2019.03.019 |
| [15] | W. E., A proposal on machine learning via dynamical systems, Communications in Mathematics and Statistics, 5, 1-11 (2017) · Zbl 1380.37154 · doi:10.1007/s40304-017-0103-z |
| [16] | Y. Gao, H. Liu, X. Wang, et al., On an artificial neural network for inverse scattering problems, Journal of Computational Physics, 448 (2022), 110771, 15 pp. · Zbl 1537.78008 |
| [17] | Y. K. Gao Zhang, Machine learning based data retrieval for inverse scattering problems with incomplete data, Journal of Inverse and Ill-Posed Problems, 29, 249-266 (2021) · Zbl 1460.35283 · doi:10.1515/jiip-2019-0101 |
| [18] | Y. P. D. Guo Monk Colton, The linear sampling method for sparse small aperture data, Applicable Analysis, 95, 1599-1615 (2016) · Zbl 1351.65070 · doi:10.1080/00036811.2015.1065317 |
| [19] | E. Haber, K. Lensink, E. Treister, et al., IMEXnet: A forward stable deep neural network, Proceedings of the International Conference on Machine Learning, California, USA, (2019), 2525-2534. |
| [20] | E. Haber and L. Ruthotto, Stable architectures for deep neural networks, Inverse Problems, 34 (2018), 014004, 22 pp. · Zbl 1426.68236 |
| [21] | S. J. Hochreiter Schmidhuber, Long short-term memory, Neural Computation, 9, 1735-1780 (1997) |
| [22] | J. Jia, B. Wu, J. Peng, et al., Recursive linearization method for inverse medium scattering problems with complex mixture Gaussian error learning, Inverse Probelms, 35 (2019), 075003, 29 pp. · Zbl 1419.65099 |
| [23] | J. S. J. Jia Yue Peng, Infinite-dimensional Bayesian approach for inverse scattering problems of a fractional Helmholtz equation, Journal of Functional Analysis, 275, 2299-2332 (2018) · Zbl 1400.35196 · doi:10.1016/j.jfa.2018.08.002 |
| [24] | J. R. A. Jumper Evans Pritzel, Highly accurate protein structure prediction with AlphaFold, Nature, 596, 583-589 (2021) |
| [25] | J. Kaipio and E. Somersalo, Statistical and Computational Inverse Problems, \(1^{st}\) edition, Applied Mathematical Sciences, 160. Springer-Verlag, New York, 2005. · Zbl 1068.65022 |
| [26] | S. Kanai, Y. Fujiwara and S. Iwamura, Preventing gradient explosions in gated recurrent units, Proceedings of the International Conference on Neural Information Processing Systems, California, USA, (2017), 435-444. |
| [27] | Y. Khoo and L. Ying, SwitchNet: A neural network model for forward and inverse scattering problems, SIAM Journal on Scientific Computing, 41 (2019), A3182-A3201. · Zbl 1425.65208 |
| [28] | J. H. Li Liu, Recovering a polyhedral obstacle by a few backscattering measurements, Journal of Differential Equations, 259, 2101-2120 (2015) · Zbl 1330.78013 · doi:10.1016/j.jde.2015.03.030 |
| [29] | J. H. W. Li Liu Tsui, An inverse scattering approach for geometric body generation: A machine learning perspective, Mathematics in Engineering, 1, 800-823 (2019) · Zbl 1435.68271 · doi:10.3934/mine.2019.4.800 |
| [30] | J. Li, H. Liu and Y. Wang, Recovering an electromagnetic obstacle by a few phaseless backscattering measurements, Inverse Problems, 33 (2017), 035011, 20 pp. · Zbl 1432.78008 |
| [31] | H. M. L. Liu Petrini Rondi, Stable determination of sound-hard polyhedral scatterers by a minimal number of scattering measurements, Journal of Differential Equations, 262, 1631-1670 (2017) · Zbl 1352.74148 · doi:10.1016/j.jde.2016.10.021 |
| [32] | H. J. Liu Zou, Uniqueness in an inverse acoustic obstacle scattering problem for both sound-hard and sound-soft polyhedral scatterers, Inverse Problems, 22, 515-524 (2006) · Zbl 1094.35142 · doi:10.1088/0266-5611/22/2/008 |
| [33] | Z. Long, Y. Lu and B. Dong, PDE-Net 2.0: Learning PDEs from data with a numeric-symbolic hybrid deep network, Journal of Computational Physics, 399 (2019), 108925, 17 pp. · Zbl 1454.65131 |
| [34] | Y. Lu, A. Zhong, Q. Li, et al., Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations, Proceedings of the International Conference on Machine Learning, Stockholm, Sweden, (2018), 3276-3285. |
| [35] | P. L. W. Meng Su Yin, Solving a kind of inverse scattering problem of acoustic waves based on linear sampling method and neural network, Journal of Alexandria Engineering, 59, 1451-1462 (2020) |
| [36] | P. S. A. Rauber Fadel Falcao, Visualizing the hidden activity of artificial neural networks, IEEE Transactions on Visualization and Computer Graphics, 23, 101-110 (2017) |
| [37] | L. E. Ruthotto Haber, Deep neural networks motivated by partial differential equations, Journal of Mathematical Imaging and Vision, 62, 352-364 (2020) · Zbl 1434.68522 · doi:10.1007/s10851-019-00903-1 |
| [38] | D. A. C. Silver Huang Maddison, Mastering the game of Go with deep neural networks and tree search, Nature, 529, 484-489 (2016) |
| [39] | D. J. K. Silver Schrittwieser Simonyan, Mastering the game of Go without human knowledge, Nature, 550, 354-359 (2017) |
| [40] | C. Tallec and Y. Ollivier, Can recurrent neural networks warp time?, Proceedings of the International Conference on Learning Representations, Vancouver, Canada, (2018), 1-13. |
| [41] | Z. Y. T. Xu Zhang Luo, Frequency principle: Fourier analysis sheds light on deep neural networks, Communications in Computational Physics, 28, 1746-1767 (2020) · Zbl 1507.68279 · doi:10.4208/cicp.OA-2020-0085 |
| [42] | W. Yin, W. Yang and H. Liu, A neural network scheme for recovering scattering obstacles with limited phaseless far-field data, Journal of Computational Physics, 417 (2020), 109594, 18 pp. · Zbl 1437.68157 |
| [43] | Y. W. P. Yin Yin Meng, On a hybrid approach for recovering multiple obstacles, Communications in Computational Physics, 31, 869-892 (2022) · Zbl 1482.62044 · doi:10.4208/cicp.OA-2021-0124 |
| [44] | Y. W. P. Yin Yin Meng, The interior inverse scattering problem for a two-layered cavity using the Bayesian method, Inverse Problems and Imaging, 16, 673-690 (2022) · Zbl 1494.78012 · doi:10.3934/ipi.2021069 |
| [45] | B. Zhou, A. Khosla, A. Lapedriza, et al., Learning deep features for discriminative localization, IEEE Conference on Computer Vision and Pattern Recognition, (2016), 2921-2929. |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
