New study on neural networks: the essential order of approximation. (English) Zbl 1398.65023
Summary: For the nearly exponential type of feedforward neural networks (neFNNs), the essential order of their approximation is revealed. It is proven that for any continuous function defined on a compact set of \(\mathbb R^d\), there exist three layers of neFNNs with the fixed number of hidden neurons that attain the essential order. Under certain assumption on the neFNNs, the ideal upper bound and lower bound estimations on approximation precision of the neFNNs are provided. The obtained results not only characterize the intrinsic property of approximation of the neFNNs, but also proclaim the implicit relationship between the precision (speed) and the number of hidden neurons of the neFNNs.
MSC:
| 65D15 | Algorithms for approximation of functions |
Keywords:
essential order of approximation; nearly exponential type neural networks; modulus of smoothnessReferences:
| [1] | Attali, J.G.; Pages, G., Approximation of functions by a multilayer perceptron: A new approach, Neural networks, 10, 1069-1081, (1997) |
| [2] | Barron, A.R., Universal approximation bounds for superpositions of a sigmoidal function, IEEE transactions on information theory, 39, 930-945, (1993) · Zbl 0818.68126 |
| [3] | Cao, F.L.; Xu, Z.B., Neural network approximation for multivariate periodic functions: estimates on approximation order, Chinese journal of computers, 24, 9, 903-908, (2001) |
| [4] | Cardaliaguet, P.; Euvrard, G., Approximation of a function and its derivatives with a neural networks, Neural networks, 5, 207-220, (1992) |
| [5] | Chen, T.P., Approximation problems in system identification with neural networks, Science in China, series A, 24, 1, 1-7, (1994) |
| [6] | Chen, T.P.; Chen, H., Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to a dynamic system, IEEE transactions on neural networks, 6, 911-917, (1995) |
| [7] | Chui, C.K.; Li, X., Approximation by ridge functions and neural networks with one hidden layer, Journal of approximation theory, 70, 131-141, (1992) · Zbl 0768.41018 |
| [8] | Chui, C.K.; Li, X., Neural networks with one hidden layer, (), 7-89 |
| [9] | Cybenko, G., Approximation by superpositions of sigmoidal function, Mathematics of control, signals, and systems, 2, 303-314, (1989) · Zbl 0679.94019 |
| [10] | Ditzian, Z.; Totik, V., Moduli of smoothness, (1987), Springer-Verlag NewYork · Zbl 0666.41001 |
| [11] | Feinerman, R.P.; Newman, D.J., Polynomial approximation, (1974), Williams & Wilkins Baltimore, MD · Zbl 0309.41006 |
| [12] | Funahashi, K.I., On the approximate realization of continuous mappings by neural networks, Neural networks, 2, 183-192, (1989) |
| [13] | Hornik, K.; Stinchombe, M.; White, H., Multilayer feedforward networks are universal approximation, Neural networks, 2, 359-366, (1989) · Zbl 1383.92015 |
| [14] | Hornik, K.; Stinchombe, M.; White, H., Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks, Neural networks, 3, 551-560, (1990) |
| [15] | Jackson, D., On the approximation by trigonometric sums and polynomials, Transactions of the American mathematical society, 13, 491-515, (1912) · JFM 43.0499.02 |
| [16] | Johnen, H.; Scherer, K., On the equivalence of the \(K\)-functional and modulus of continuity and some applications, (), 119-140 · Zbl 0348.26005 |
| [17] | Kůrkova, V.; Kainen, P.C.; Kreinovich, V., Estimates for the number of hidden units and variation with respect to half-space, Neural networks, 10, 1068-1078, (1997) |
| [18] | Leshno, M.; Lin, V.Y.; Pinks, A.; Schocken, S., Multilayer feedforward networks with a nonpolynomial activation function can approximate any function, Neural networks, 6, 861-867, (1993) |
| [19] | Maiorov, V.; Meir, R.S., Approximation bounds for smooth functions in \(C(\mathcal{R}^d)\) by neural and mixture networks, IEEE transactions on neural networks, 9, 969-978, (1998) |
| [20] | Mhaskar, H.N., Neural networks for optimal approximation for smooth and analytic functions, Neural computation, 8, 164-177, (1996) |
| [21] | Mhaskar, H.N.; Khachikyan, L., Neural networks for functions approximation, (), 21-29 |
| [22] | Mhaskar, H.N.; Micchelli, C.A., Approximation by superposition of a sigmoidal function, Advances in applied mathematics, 13, 350-373, (1992) · Zbl 0763.41015 |
| [23] | Mhaskar, H.N.; Micchelli, C.A., Dimension-independent bounds on the degree of approximation by neural networks, IBM journal of research and development, 38, 277-284, (1994) · Zbl 0823.41012 |
| [24] | Mhaskar, H.N.; Micchelli, C.A., Degree of approximation by neural networks with a single hidden layer, Advances in applied mathematics, 16, 151-183, (1995) · Zbl 0885.42012 |
| [25] | Nikol’skii, S.M., Approximation of functions of several variables and imbedding theorems, (1975), Springer Berlin, Heidelberg, New York · Zbl 0307.46024 |
| [26] | Ritter, G. (1994). Jackson’s theorems and the number of hidden units in Neural networks for uniform approximation. Technical Report, MIP9415. Univ. Passau, Fak. Math. Inform..; Ritter, G. (1994). Jackson’s theorems and the number of hidden units in Neural networks for uniform approximation. Technical Report, MIP9415. Univ. Passau, Fak. Math. Inform.. |
| [27] | Ritter, G., Efficient estimation of neural weights by polynomial approximation, IEEE transactions on information theory, 45, 1541-1550, (1999) · Zbl 0952.65026 |
| [28] | Soardi, P.M., () |
| [29] | Suzuki, S., Constructive function approximation by three-layer artificial neural networks, Neural networks, 11, 1049-1058, (1998) |
| [30] | Xu, Z.B.; Cao, F.L., The essential order of approximation for neural networks, Science in China, series F, 47, 97-112, (2004) · Zbl 1186.82063 |
| [31] | Xu, Z.B.; Wang, J.J., The essential order of approximation for neural networks, Science in China, series F. information sciences, 49, 4, 446-460, (2006) · Zbl 1121.41033 |
| [32] | Yoshifusa, I., Approximation of functions on a compact set by finite sums of Sigmoid function without scaling, Neural networks, 4, 817-826, (1991) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
