Local extreme learning machines and domain decomposition for solving linear and nonlinear partial differential equations. (English) Zbl 1507.65174
Summary: We present a neural network-based method for solving linear and nonlinear partial differential equations, by combining the ideas of extreme learning machines (ELM), domain decomposition and local neural networks. The field solution on each sub-domain is represented by a local feed-forward neural network, and \(C^k\) continuity conditions are imposed on the sub-domain boundaries. Each local neural network consists of a small number of hidden layers, while its last hidden layer can be wide. The weight/bias coefficients in all the hidden layers of the local neural networks are pre-set to random values and fixed throughout the computation, and only the weight coefficients in the output layers of the local neural networks are training parameters. The overall neural network is trained by a linear or nonlinear least squares computation, not by the back-propagation type algorithms. We introduce a block time-marching scheme together with the presented method for long-time simulations of time-dependent linear/nonlinear partial differential equations. The current method exhibits a clear sense of convergence with respect to the degrees of freedom in the neural network. Its numerical errors typically decrease exponentially or nearly exponentially as the number of training parameters, or the number of training data points, or the number of sub-domains increases. Extensive numerical experiments have been performed to demonstrate the computational performance of the presented method. We also demonstrate its capability for long-time dynamic simulations with some test problems. We compare the presented method with the deep Galerkin method (DGM) and the physics-informed neural network (PINN) method in terms of the accuracy and computational cost. The current method exhibits a clear superiority, with its numerical errors and network training time considerably smaller (typically by orders of magnitude) than those of DGM and PINN. We also compare the current method with the classical finite element method (FEM). The computational performance of the current method is on par with, and often exceeds, the FEM performance in terms of the accuracy and computational cost.
MSC:
| 65M55 | Multigrid methods; domain decomposition for initial value and initial-boundary value problems involving PDEs |
| 65N55 | Multigrid methods; domain decomposition for boundary value problems involving PDEs |
Keywords:
local extreme learning machine; extreme learning machine; neural network; least squares; nonlinear least squares; domain decompositionReferences:
| [1] | Goodfellow, I.; Bengio, Y.; Courville, A., Deep Learning (2016), The MIT Press · Zbl 1373.68009 |
| [2] | Hornik, K.; Stinchcombe, M.; White, H., Multilayer feedforward networks are universal approximators, Neural Netw., 2, 359-366 (1989) · Zbl 1383.92015 |
| [3] | Hornik, K.; Stinchcombe, M.; White, H., Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks, Neural Netw., 3, 551-560 (1990) |
| [4] | Cotter, N. E., The Stone-Weierstrass theorem and its application to neural networks, IEEE Trans. Neural Netw., 4, 290-295 (1990) |
| [5] | Li, X., Simultaneous approximations of mulvariate functions and their derivatives by neural networks with one hidden layer, Neurocomputiing, 12, 327-343 (1996) · Zbl 0861.41013 |
| [6] | Werbos, P. J., Beyond Regression: New Tools for Prediction and Alaysis in the Behavioral Sciences (1974), (Ph.D. thesis) |
| [7] | Haykin, S., Neural Networks: A Comprehensive Foundation (1999), Prentice Hall · Zbl 0934.68076 |
| [8] | Sirignano, J.; Spoliopoulos, K., DGM: A deep learning algorithm for solving partial differential equations, J. Comput. Phys., 375, 1339-1364 (2018) · Zbl 1416.65394 |
| [9] | Raissi, M.; Perdikaris, P.; Karniadakis, G. E., Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys., 378, 686-707 (2019) · Zbl 1415.68175 |
| [10] | Lagaris, I. E.; Likas, A. C.; Fotiadis, D. I., Artificial neural networks for solving ordinary and partial differential equations, IEEE Trans. Neural Netw., 9, 987-1000 (1998) |
| [11] | Lagaris, I. E.; Likas, A. C.; Papageorgiou, D. G., Neural-network methods for boundary value problems with irregular boundaries, IEEE Trans. Neural Netw., 11, 1041-1049 (2000) |
| [12] | Rudd, K.; Ferrari, S., A constrained integration (CINT) approach to solving partial differential equations using artificial neural networks, Neurocomputing, 155, 277-285 (2015) |
| [13] | E, W.; Yu, B., The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems, Commun. Math. Stat., 6, 1-12 (2018) · Zbl 1392.35306 |
| [14] | Winovich, N.; Ramani, K.; Lin, G., ConvPDE-UQ: Convolutional neural networks with quantified uncertainty for heterogeneous elliptic partial differential equations on varied domains, J. Comput. Phys., 394, 263-279 (2019) · Zbl 1457.65245 |
| [15] | He, J.; Xu, J., Mgnet: A unified framework for multigrid and convolutional neural network, Sci. China Math., 62, 1331-1354 (2019) · Zbl 1476.65026 |
| [16] | Xing, W.; Kirby, R. M.; Zhe, S., Deep corgionalization for the emulation of spatial-temporal fields (2019), ArXiv:1910.07577 |
| [17] | Zang, Y.; Bao, G.; Ye, X.; Zhou, H., Weak adversarial networks for high-dimensional partial differential equations, J. Comput. Phys., 411, Article 109409 pp. (2020) · Zbl 1436.65156 |
| [18] | Lin, Y. Wang G., Efficient deep learning techniques for multiphase flow simulation in heterogeneous porous media, J. Comput. Phys., 401, Article 108968 pp. (2020) |
| [19] | Samanaiego, E.; Anitescu, C.; Goswami, S.; Nguyen-Thanh, V. M.; Guo, H.; Hamdia, K.; Zhuang, X.; Rabczuk, T., An energy approach to the solution of partial differential equations in computational mechanics via machine learning: concepts, implementation and applications, Comput. Methods Appl. Mech. Engrg., 362, Article 112790 pp. (2020) · Zbl 1439.74466 |
| [20] | Xu, J., The finite neuron method and convergence analysis (2020), ArXiv:2010.01458 |
| [21] | Baydin, A. G.; Pearlmutter, B. A.; Radul, A. A.; Siskind, J. M., Automatic differentiation in machine learning: a survey, J. Mach. Learn. Res., 18, 1-43 (2018) · Zbl 06982909 |
| [22] | Jagtap, A. D.; Kharazmi, E.; Karniadakis, G. E., Conservative physics-informed neural networks on discrete domains for conservation laws: applications to forward and inverse problems, Comput. Methods Appl. Mech. Engrg., 365, Article 113028 pp. (2020) · Zbl 1442.92002 |
| [23] | Huang, G.-B.; Zhu, Q.-Y.; Siew, C.-K., Extreme learning machine: theory and applications, Neurocomputing, 70, 489-501 (2006) |
| [24] | Huang, G.-B.; Wang, D. H.; Lan, Y., Extreme learning machines: a survey, Int. J. Mach. Learn. Cybern., 2, 107-122 (2011) |
| [25] | Golub, G. H.; Loan, C. F.V., Matrix Computations (1996), Johns Hopkins Press, MD · Zbl 0865.65009 |
| [26] | Pao, Y. H.; Park, G. H.; Sobajic, D. J., Learning and generalization characteristics of the random vector functional-link net, Neurocomputing, 6, 163-180 (1994) |
| [27] | Igelnik, B.; Pao, Y. H., Stochastic choice of basis functions in adaptive function approximation and the functional-link net, IEEE Trans. Neural Netw., 6, 1320-1329 (1995) |
| [28] | Maass, W.; Markram, H., On the computational power of recurrent circuits of spiking neurons, J. Comput. System Sci., 69, 593-616 (2004) · Zbl 1076.68062 |
| [29] | Jaeger, H.; Lukosevicius, M.; Popovici, D.; Siewert, U., Optimization and applications of echo state networks with leaky integrator neurons, Neural Netw., 20, 335-352 (2007) · Zbl 1132.68554 |
| [30] | Zhang, L.; Suganthan, P. N., A comprehensive evaluation of random vector functional link networks, Inform. Sci., 367-368, 1094-1105 (2016) |
| [31] | Webster, C. S., Alan Turing’s unorganized machines and artificial neural networks: his remarkable early work and future possibilities, Evol. Intel., 5, 35-43 (2012) |
| [32] | Rosenblatt, F., The perceptron: a probabilistic model for information storage and organization in the brain, Psychol. Rev., 65, 386-408 (1958) |
| [33] | Balasundaram, S.; Kapil, Application of error minimized extreme learning machine for simultaneous learning of a function and its derivativs, Neurocomputing, 74, 2511-2519 (2011) |
| [34] | Yang, Y.; Hou, M.; Luo, J., A novel improved extreme learning machine algorithm in solving ordinary differential equations by Legendre neural network methods, Adv. Differential Equations, 469, 1-24 (2018) · Zbl 1448.68382 |
| [35] | Sun, H.; Hou, M.; Yang, Y.; Zhang, T.; Weng, F.; Han, F., Solving partial differential equations based on Bernsteirn neural network and extreme learning machine algorithm, Neural Process. Lett., 50, 1153-1172 (2019) |
| [36] | Panghal, S.; Kumar, M., Optimization free neural network approach for solving ordinary and partial differential equations, Eng. Comput. (2020), Early Access |
| [37] | Liu, H.; Xing, B.; Wang, Z.; Li, L., Legendre neural network method for several classes of singularly perturbed differential equations based on mapping and piecewise optimization technology, Neural Process. Lett., 51, 2891-2913 (2020) |
| [38] | Dwivedi, V.; Srinivasan, B., Physics informed extreme learning machine (PIELM) \(-\) a rapid method for the numerical solution of partial differential equations, Neurocomputing, 391, 96-118 (2020) |
| [39] | Smith, Barry F.; Bjø rstad, Petter E.; Gropp, William D., Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations (1996), Cambridge University Press · Zbl 0857.65126 |
| [40] | Toselli, A.; Widlund, O., Domain Decomposition Methods \(-\) Algorithms and Theory (2005), Springer · Zbl 1069.65138 |
| [41] | Dong, S., BDF-like methods for nonlinear dynamic analysis, J. Comput. Phys., 229, 3019-3045 (2010) · Zbl 1307.74069 |
| [42] | Dong, S.; Shen, J., A pressure correction scheme for generalized form of energy-stable open boundary conditions for incompressible flows, J. Comput. Phys., 291, 254-278 (2015) · Zbl 1349.76198 |
| [43] | Dong, S., Multiphase flows of N immiscible incompressible fluids: a reduction-consistent and thermodynamically-consistent formulation and associated algorithm, J. Comput. Phys., 361, 1-49 (2018) · Zbl 1391.76804 |
| [44] | Li, K.; Tang, K.; Wu, T.; Liao, Q., D3M: A deep domain decomposition method for partial differential equations, IEEE Access, 8, 5283-5294 (2020) |
| [45] | Karniadakis, G. E.; Sherwin, S. J., Spectral/Hp Element Methods for Computational Fluid Dynamics (2005), Oxford University Press · Zbl 1116.76002 |
| [46] | Yu, Y.; Kirby, R. M.; Karniadakis, G. E., Spectral element and hp methods, Encycl. Comput. Mech., 1, 1-43 (2017) |
| [47] | Zheng, X.; Dong, S., An eigen-based high-order expansion basis for structured spectral elements, J. Comput. Phys., 230, 8573-8602 (2011) · Zbl 1352.76087 |
| [48] | Dong, S.; Yosibash, Z., A parallel spectral element method for dynamic three-dimensional nonlinear elasticity problems, Comput. Struct., 87, 59-72 (2009) |
| [49] | Dong, S.; Shen, J., A time-stepping scheme involving constant coefficient matrices for phase field simulations of two-phase incompressible flows with large density ratios, J. Comput. Phys., 231, 5788-5804 (2012) · Zbl 1277.76118 |
| [50] | Kingma, D. P.; Ba, J., Adam: a method for stochastic optimization (2014), ArXiv:1412.6980 |
| [51] | Nocedal, J.; Wright, S. J., Numerical Optimization (2006), Springer · Zbl 1104.65059 |
| [52] | Langtangen, H. P.; Logg, A., Solving PDEs in Python, the FEniCS Tutorial I (2016), SpringerOpen · Zbl 1376.65144 |
| [53] | Dong, S.; Ni, N., A method for representing periodic functions and enforcing exactly periodic boundary conditions with deep neural networks, J. Comput. Phys., 435, Article 110242 pp. (2021) · Zbl 07503727 |
| [54] | Courant, R. L., Variational methods for the solution of problems of equilibrium and vibration, Bull. Amer. Math. Soc., 49, 1-23 (1943) · Zbl 0063.00985 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
