A recursive algorithm for nonlinear least-squares problems. (English) Zbl 1144.62080
Summary: The solution of nonlinear least-squares problems is investigated. The asymptotic behavior is studied and conditions for convergence are derived. To deal with such problems in a recursive and efficient way an algorithm is proposed that is based on a modified extended Kalman filter (MEKF). The error of the MEKF algorithm is proved to be exponentially bounded. Batch and iterated versions of the algorithm are given, too. As an application, the algorithm is used to optimize the parameters in certain nonlinear input-output mappings. Simulation results on interpolation of real data and prediction of chaotic time series are shown.
MSC:
| 62M20 | Inference from stochastic processes and prediction |
| 37M10 | Time series analysis of dynamical systems |
| 90C30 | Nonlinear programming |
| 65C60 | Computational problems in statistics (MSC2010) |
| 90C90 | Applications of mathematical programming |
Keywords:
Nonlinear programming; Nonlinear least squares; Extended Kalman filter; Recursive optimization; Batch algorithmsSoftware:
Neural Network ToolboxReferences:
| [1] | Bates, D.M., Watts, D.G.: Nonlinear Regression and Its Applications. Wiley, New York (1988) · Zbl 0728.62062 |
| [2] | Bertsekas, D.P.: Nonlinear Programming, 2nd edn. Athena Scientific, Belmont (1999) |
| [3] | Bertsekas, D.P.: Incremental least–squares methods and the extended Kalman filter. SIAM J. Optim. 6(3), 807–822 (1996) · Zbl 0945.93026 · doi:10.1137/S1052623494268522 |
| [4] | Feldkamp, L.A., Prokhorov, D.V., Eagen, C.F., Yuan, F.: Enhanced multi-stream Kalman filter training for recurrent networks. In: Suykens, J., Vandewalle, J. (eds.) Nonlinear Modeling: Advanced Black-Box Techniques, pp. 29–53. Kluwer Academic, Dordrecht (1998) |
| [5] | Moriyama, H., Yamashita, N., Fukushima, M.: The incremental Gauss–Newton algorithm with adaptive stepsize rule. Comput. Optim. Appl. 26(2), 107–141 (2003) · Zbl 1081.90050 · doi:10.1023/A:1025703629626 |
| [6] | Shuhe, H.: Consistency for the least squares estimator in nonlinear regression model. Stat. Probab. Lett. 67(2), 183–192 (2004) · Zbl 1058.62025 · doi:10.1016/j.spl.2003.11.020 |
| [7] | Reif, K., Günter, S., Yaz, E., Unbehauen, R.: Stochastic stability of the discrete-time extended Kalman filter. IEEE Trans. Autom. Control 44(4), 714–728 (1999) · Zbl 0967.93090 · doi:10.1109/9.754809 |
| [8] | Kurková, V., Sanguineti, M.: Learning with generalization capability by kernel methods of bounded complexity. J. Complex. 21(3), 350–367 (2005) · Zbl 1095.68044 · doi:10.1016/j.jco.2004.11.002 |
| [9] | Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representation by error propagation. In: Rumelhart, D.E., McClelland, J.L., PDP Research Group (eds.) Parallel Distributed Processing: Explorations in the Microstructures of Cognition, vol. I: Foundations, pp. 318–362. MIT, Cambridge (1986) |
| [10] | Widrow, B., Lehr, M.A.: 30 years of adaptive neural networks: perceptron, madaline, and backpropagation. Proc. IEEE 78(9), 1415–1442 (1990) · doi:10.1109/5.58323 |
| [11] | Ortega, J.M.: Numerical Analysis: A Second Course. SIAM, Philadelphia (1990). Reprint of the 1972 edition by Academic, Now York · Zbl 0701.65002 |
| [12] | Jazwinski, A.H.: Stochastic Processes and Filtering Theory. Academic, New York (1970) · Zbl 0203.50101 |
| [13] | Pinkus, A.: Approximation theory of the MLP model in neural networks. Acta Numer. 8, 143–196 (1999) · Zbl 0959.68109 · doi:10.1017/S0962492900002919 |
| [14] | Park, J., Sandberg, I.W.: Universal approximation using radial–basis–function networks. Neural Comput. 3(2), 246–257 (1991) · doi:10.1162/neco.1991.3.2.246 |
| [15] | Heskes, T., Wiegerinck, W.: A theoretical comparison of batch-mode, on-line, cyclic, and almost-cyclic learning. IEEE Trans. Neural Netw. 7(4), 919–925 (1996) · doi:10.1109/72.508935 |
| [16] | Demuth, H., Beale, M.: Neural Network Toolbox–User’s Guide. The Math Works, Natick (2000) |
| [17] | Bell, B.M., Cathey, F.W.: The iterated Kalman filter update as a Gauss–Newton method. IEEE Trans Autom. Control 38(2), 294–297 (1993) · Zbl 0775.93237 · doi:10.1109/9.250476 |
| [18] | Fletcher, R.: Practical Methods of Optimization. Wiley, Chichester (1987) · Zbl 0905.65002 |
| [19] | Battiti, R.: First- and second-order methods for learning: between steepest descent and Newton’s method. Neural Comput. 4(2), 141–166 (1992) · doi:10.1162/neco.1992.4.2.141 |
| [20] | Tollenaere, T.: SuperSAB: fast adaptive backpropagation with good scaling properties. Neural Netw. 3(5), 561–573 (1990) · doi:10.1016/0893-6080(90)90006-7 |
| [21] | Jacobs, R.A.: Increased rates of convergence through learning rate adaption. Neural Netw. 1(4), 295–307 (1988) · doi:10.1016/0893-6080(88)90003-2 |
| [22] | Denton, J.W., Hung, M.S.: A comparison of nonlinear optimization methods for supervised learning in multilayer feedforward neural networks. Eur. J. Oper. Res. 93(2), 358–368 (1996) · Zbl 0912.90253 · doi:10.1016/0377-2217(96)00035-5 |
| [23] | Stinchcombe, M., White, H.: Approximation and learning unknown mappings using multilayer feedforward networks with bounded weights. In: Proc. Int. Joint Conf. on Neural Networks IJCNN’90, pp. III7–III16 (1990) |
| [24] | Singhal, S., Wu, L.: Training multilayer perceptrons with the extended Kalman algorithm. In: Touretzky, D.S. (ed.) Advances in Neural Information Processing Systems 1, pp. 133–140. Morgan Kaufmann, San Mateo (1989) |
| [25] | Ruck, D.W., Rogers, S.K., Kabrisky, M., Maybeck, P.S., Oxley, M.E.: Comparative analysis of backpropagation and the extended Kalman filter for training multilayer perceptrons. IEEE Trans. Pattern Anal. Mach. Intell. 14(6), 686–691 (1992) · doi:10.1109/34.141559 |
| [26] | Iiguni, Y., Sakai, H., Tokumaru, H.: A real-time learning algorithm for a multilayered neural network based on the extended Kalman filter. IEEE Trans. Signal Process. 40(4), 959–966 (1992) · doi:10.1109/78.127966 |
| [27] | Schottky, B., Saad, D.: Statistical mechanics of EKF learning in neural networks. J. Phys. A: Math. Gen. 32(9), 1605–1621 (1999) · Zbl 0964.82044 · doi:10.1088/0305-4470/32/9/009 |
| [28] | Nishiyama, K., Suzuki, K.: H learning of layered neural networks. IEEE Trans. Neural Netw. 12(6), 1265–1277 (2001) · doi:10.1109/72.963763 |
| [29] | Leung, C.-S., Tsoi, A.-C., Chan, L.W.: Two regularizers for recursive least squared algorithms in feedforward multilayered neural networks. IEEE Trans. Neural Netw. 12(6), 1314–1332 (2001) · doi:10.1109/72.963768 |
| [30] | Alessandri, A., Sanguineti, M., Maggiore, M.: Optimization-based learning with bounded error for feedforward neural networks. IEEE Trans. Neural Netw. 13(2), 261–273 (2002) · doi:10.1109/72.991413 |
| [31] | Prechelt, L.: PROBEN 1–A set of neural network benchmark problems and benchmarking rules. Tech. Rep. 21/94, Fakultät für Informatik, Universität Karlsruhe, Germany, September 1994, Anonymous FTP: /pub/papers/techreports/1994/1994-21.ps.gzonftp.ira.uka.de |
| [32] | Mackey, M.C., Glass, L.: Oscillation and chaos in physiological control systems. Science 197, 287–289 (1977) · Zbl 1383.92036 · doi:10.1126/science.267326 |
| [33] | Hardy, G., Littlewood, J.E., Polya, G.: Inequalities. Cambridge University Press, Cambridge (1989) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
