Abstract
In this paper, by constructing a generalized Armijo search method, a novel conjugate gradient (CG) model has been proposed to training a common three-layer backpropagation (BP) neural network. Compared with the classical gradient descent method, this algorithm efficiently accelerates the convergence speed due to the existence of the additional conjugate direction. Essentially, the optimal learning rate of each epoch is determined by the given inexact line search strategy. The presented model does not significantly increase the computational cost in dealing with real applications. Two benchmark simulations have been performed to illustrate the promising advantages of the proposed algorithm.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Xia, Y., Hu, R.: Fuzzy neural network based energy efficiencies control in the heating energy supply system responding to the changes of user demands. J. Netw. Intell. 2, 186–194 (2017)
Erkaymaz, O., Şenyer, İ., Uzun, R.: Detection of knee abnormality from surface EMG signals by artificial neural networks. In: 2017 25th Signal Processing and Communications Applications Conference (SIU) (2017)
Li, X.D., Rakkiyappan, R.: Impulsive controller design for exponential synchronization of chaotic neural networks with mixed delays. Commun. Nonlinear Sci. Numer. Simul. 18, 1515–1523 (2013)
Li, X.D., Rakkiyappan, R., Velmurugan, G.: Dissipativity analysis of memristor-based complex-valued neural networks with time-varying delays. Inf. Sci. 294, 645–665 (2015)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323, 533–536 (1986)
Park, D.C., Elsharkawi, M.A., Marks, R.J., Atlas, L.E., Damborg, M.J.: Electric-load forecasting using an artificial neural networks. IEEE Trans. Power Syst. 6, 442–449 (1991)
Saini, L.M., Soni, M.K.: Artificial neural network-based peak load forecasting using conjugate gradient methods. IEEE Trans. Power Syst. 17, 907–912 (2002)
Luo, B., Wu, H.N., Li, H.X.: Adaptive optimal control of highly dissipative nonlinear spatially distributed processes with neuro-dynamic programming. IEEE Trans. Neural Netw. Learn. Syst. 26, 684–696 (2015)
Luo, B., Liu, D.R., Wu, H.N., Wang, D., Lewis, F.L.: Policy gradient adaptive dynamic programming for data-based optimal control. IEEE Trans. Cybern. 99, 1–14 (2016)
Zhang, H.S., Tang, Y.L.: Online gradient method with smoothing \(\ell _0\) regularization for feedforward neural networks. Neurocomputing 224, 1–8 (2017)
Lu, C.N., Wu, H.T., Vemuri, S.: Neural network based short-term load forecasting. IEEE Trans. Power Syst. 8, 336–342 (1993)
Papalexopoulos, A.D., Hao, S.Y., Peng, T.M.: An implementation of a neural-network-based load forecasting-model for the EMS. IEEE Trans. Power Syst. 9, 1956–1962 (1994)
Zhang, H.S., Mandic, D.P.: Is a complex-valued stepsize advantageous in complex-valued gradient learning algorithms? IEEE Trans. Neural Netw. Learn. Syst. 27, 2730–2735 (2016)
Goodband, J.H., Haas, O.C.L., Mills, J.A.: A comparison of neural network approaches for on-line prediction in IGRT. Med. Phys. 35, 1113–1122 (2008)
Hagan, M.T., Demuth, H.B., Beale, M.H.: Neural Network Design. PWS Publisher, Boston (1996)
Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (2006). doi:10.1007/978-0-387-40065-5
Hestenes, M.R., Stiefel, E.L.: Method of conjugate gradients for solving linear systems. National Bureau of Standards, Washington (1952)
Fletcher, R., Reeves, C.M.: Function minimization by conjugate gradients. Comput. J. 7, 149–154 (1964)
Dai, Y.H., Yuan, Y.X.: Nonlinear Conjugate Gradient Methods. Shanghai Scientific, Technical Publishers, Shanghai (2000)
Gonzalez, A., Dorronsoro, J.R.: Natural conjugate gradient training of multilayer perceptrons. Neurocomputing 71, 2499–2506 (2008)
Sun, Q.Y., Liu, X.H.: Global convergence results of a new three terms conjugate gradient method with generalized Armijo step size rule. Mathematica Numerica Sinica 26, 25–36 (2004)
Polak, E., Ribiere, G.: Note sur la convergence de directions conjugates. Revue Francaise d’Informatique et de Recherche Operationnelle 16, 94–112 (1969)
Polyak, B.T.: The conjugate gradient method in extremal problems. USSR Comput. Math. Math. Phys. 9, 94–112 (1969)
Shen, X.Z., Shi, X.Z., Meng, G.: Online algorithm of blind source separation based on conjugate gradient method. Circuits Syst. Sig. Process. 25, 381–388 (2006)
Wang, J., Wu, W., Zurada, J.M.: Deterministic convergence of conjugate gradient method for feedforward neural networks. Neurocomputing 74, 2368–2376 (2011)
Magoulas, G.D., Vrahatis, M.N., Androulakis, G.S.: Effective backpropagation training with variable stepsize. Neural Netw. 10, 69–82 (1997)
Orozco-Henao, C., Bretas, A.S., Chouhy-Leborgne, R., Herrera-Orozco, A.R., Marín-Quintero, J.: Active distribution network fault location methodology: a minimum fault reactance and Fibonacci search approach. Electr. Power Energy Syst. 84, 232–241 (2017)
Vieira, D.A.G., Lisboa, A.C.: Line search methods with guaranteed asymptotical convergence to an improving local optimum of multimodal functions. Eur. J. Oper. Res. 235, 38–46 (2014)
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China (No. 61305075, 11401185), the China Postdoctoral Science Foundation (No. 2012M520624), the Natural Science Foundation of Shandong Province (Nos. ZR2013FQ004, ZR2013DM015, ZR2015AL014), the Specialized Research Fund for the Doctoral Program of Higher Education of China (No. 20130133120014) and the Fundamental Research Funds for the Central Universities (Nos. 13CX05016A, 14CX05042A, 15CX05053A, 15CX08011A, 15CX02064A).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Zhang, B., Gao, T., Li, L., Sun, Z., Wang, J. (2017). An Improved Conjugate Gradient Neural Networks Based on a Generalized Armijo Search Method. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, ES. (eds) Neural Information Processing. ICONIP 2017. Lecture Notes in Computer Science(), vol 10637. Springer, Cham. https://doi.org/10.1007/978-3-319-70093-9_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-70093-9_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-70092-2
Online ISBN: 978-3-319-70093-9
eBook Packages: Computer ScienceComputer Science (R0)