A new structured spectral conjugate gradient method for nonlinear least squares problems. (English) Zbl 07925698
Summary: Least squares models appear frequently in many fields, such as data fitting, signal processing, machine learning, and especially artificial intelligence. Nowadays, the model is a popular and sophisticated way to make predictions about real-world problems. Meanwhile, conjugate gradient methods are traditionally known as efficient tools to solve unconstrained optimization problems, especially in high-dimensional cases. This paper presents a new structured spectral conjugate gradient method based on a modification of the modified structured secant equation of Zhang, Xue, and Zhang. The proposed method uses a novel appropriate spectral parameter. It is proved that the new direction satisfies the sufficient descent condition regardless of the line search. The global convergence of the proposed method is demonstrated under some standard assumptions. Numerical experiments show that our proposed method is efficient and can compete with other existing algorithms in this area.
MSC:
| 65-XX | Numerical analysis |
| 90C53 | Methods of quasi-Newton type |
| 49M37 | Numerical methods based on nonlinear programming |
| 65K05 | Numerical mathematical programming methods |
Keywords:
nonlinear least squares problems; structured spectral conjugate gradient method; modified secant equation; sufficient descent condition; global convergenceReferences:
| [1] | Al-Baali, M.; Fletcher, R., Variational methods for non-linear least-squares, J. Oper. Res. Soc., 36, 405-421, 1985 · Zbl 0578.65064 |
| [2] | Amini, K.; Rizi, AG, A new structured quasi-Newton algorithm using partial information on Hessian, J. Comput. Appl. Math., 234, 805-811, 2010 · Zbl 1190.65094 |
| [3] | Aminifard, Z., Babaie-Kafaki, S.: An optimal parameter choice for the Dai-Liao family of conjugate gradient methods by avoiding a direction of the maximum magnification by the search direction matrix. 4OR 17, 317-330 (2019) · Zbl 1425.90134 |
| [4] | Aminifard, Z.; Babaie-Kafaki, S., Modified spectral conjugate gradient methods based on the quasi-Newton aspects, Pacific. J. Optim., 16, 581-594, 2020 · Zbl 1457.90171 |
| [5] | Amini, K.; Faramarzi, P.; Pirfalah, N., A modified Hestenes-Stiefel conjugate gradient method with an optimal property, Optim. Methods. Softw., 34, 770-782, 2019 · Zbl 1461.65114 |
| [6] | Andrei, N., Scaled conjugate gradient algorithms for unconstrained optimization, Comput. Optim. Appl., 38, 401-416, 2007 · Zbl 1168.90608 |
| [7] | Andrei, N., New accelerated conjugate gradient algorithms as a modification of Dai-Yuan’s computational scheme for unconstrained optimization, J. Comput. Appl. Math., 234, 3397-3410, 2010 · Zbl 1407.65060 |
| [8] | Babaie-Kafaki, S., A modified scaling parameter for the memoryless BFGS updating formula, Numer. Algorithms, 72, 425-433, 2016 · Zbl 1342.90227 |
| [9] | Babaie-Kafaki, S., Ghanbari, R.: A class of adaptive Dai-Liao conjugate gradient methods based on the scaled memoryless BFGS update. 4OR 15, 85-92 (2017) · Zbl 1360.90293 |
| [10] | Beck, A.: Introduction to nonlinear optimization: theory, algorithms, and applications with MATLAB. SIAM (2014) · Zbl 1320.90001 |
| [11] | Birgin, EG, Martínez JM,: A spectral conjugate gradient method for unconstrained optimization, Appl. Math. Optim., 43, 117-128, 2001 · Zbl 0990.90134 |
| [12] | Dai, YH; Kou, CX, A nonlinear conjugate gradient algorithm with an optimal property and an improved Wolfe line search, SIAM J. Optim., 23, 296-320, 2013 · Zbl 1266.49065 |
| [13] | Dai, YH; Yuan, Y., A nonlinear conjugate gradient method with a strong global convergence property, SIAM J. Optim., 10, 177-182, 1999 · Zbl 0957.65061 |
| [14] | Dehghani, R.; Mahdavi-Amiri, N., Scaled nonlinear conjugate gradient methods for nonlinear least squares problems, Numer. Algorithms, 82, 1-20, 2019 · Zbl 1421.90166 |
| [15] | Dennis, JE; Gay, DM; Walsh, RE, An adaptive nonlinear least-squares algorithm, ACM Trans. Math. Softw., 7, 348-368, 1981 · Zbl 0464.65040 |
| [16] | Dennis, JE; Martinez, HJ; Tapia, RA, Convergence theory for the structured BFGS secant method with an application to nonlinear least squares, J. Optim. Theory Appl., 61, 161-178, 1989 · Zbl 0645.65026 |
| [17] | Dennis, JE; Walker, HF, Convergence theorems for least-change secant update methods, SIAM J. Numer. Anal., 18, 949-987, 1981 · Zbl 0527.65032 |
| [18] | Dolan, ED; Moré, JJ, Benchmarking optimization software with performance profiles, Math. Program., 91, 201-213, 2002 · Zbl 1049.90004 |
| [19] | Engels, JR, Martínez HJ,: Local and superlinear convergence for partially known quasi-Newton methods, SIAM J. Optim., 1, 42-56, 1991 · Zbl 0752.90063 |
| [20] | Faramarzi, P.; Amini, K., A scaled three-term conjugate gradient method for large-scale unconstrained optimization problem, Calcolo, 56, 1-15, 2019 · Zbl 1433.90160 |
| [21] | Faramarzi, P.; Amini, K., A modified spectral conjugate gradient method with global convergence, J. Optim. Theory Appl., 182, 667-690, 2019 · Zbl 1422.90053 |
| [22] | Faramarzi, P., Amini, K.: A spectral three-term Hestenes-Stiefel conjugate gradient method. 4OR 19, 71-92 (2021) · Zbl 1471.90138 |
| [23] | Fletcher, R.; Reeves, CM, Function minimization by conjugate gradients, Comput. J., 7, 149-154, 1964 · Zbl 0132.11701 |
| [24] | Hager, WW; Zhang, H., A new conjugate gradient method with guaranteed descent and an efficient line search, SIAM J. Optim., 16, 170-192, 2005 · Zbl 1093.90085 |
| [25] | Hartley, HO, The modified Gauss-Newton method for the fitting of non-linear regression functions by least squares, Technometrics, 3, 269-280, 1961 · Zbl 0096.34603 |
| [26] | Hestenes, MR; Stiefel, E., Methods of conjugate gradients for solving linear systems, J. Res. Natl. Bur. Stand., 49, 409-436, 1952 · Zbl 0048.09901 |
| [27] | Huschens, J., On the use of product structure in secant methods for nonlinear least squares problems, SIAM J. Optim., 4, 108-129, 1994 · Zbl 0798.65064 |
| [28] | Ibrahim, SM; Yakubu, UA; Mamat, M., Application of spectral conjugate gradient methods for solving unconstrained optimization problems, Int. J. Optim. Control Theor. Appl., 10, 198-205, 2020 |
| [29] | Jian, J.; Chen, Q.; Jiang, X., A new spectral conjugate gradient method for large-scale unconstrained optimization, Optim. Methods Softw., 32, 503-515, 2017 · Zbl 1365.90201 |
| [30] | Kobayashi, M.; Narushima, Y.; Yabe, H., Nonlinear conjugate gradient methods with structured secant condition for nonlinear least squares problems, J. Comput. Appl. Math., 234, 375-397, 2010 · Zbl 1223.65043 |
| [31] | Levenberg, K., A method for the solution of certain non-linear problems in least squares, Q. Appl. Math., 2, 164-168, 1944 · Zbl 0063.03501 |
| [32] | Li, M., A family of three-term nonlinear conjugate gradient methods close to the memoryless BFGS method, Optim. Lett., 12, 1911-1927, 2018 · Zbl 1412.90165 |
| [33] | Liu, H.; Yao, Y.; Qian, XY; Wang, H., Some nonlinear conjugate gradient methods based on spectral scaling secant equations, Comput. Appl. Math., 35, 639-651, 2016 · Zbl 1376.90062 |
| [34] | Livieris, IE; Pintelas, P., A new class of spectral conjugate gradient methods based on a modified secant equation for unconstrained optimization, J. Comput. Appl. Math., 239, 396-405, 2013 · Zbl 1258.65058 |
| [35] | Luksan, L., Vlcek, J.: Sparse and partially separable test problems for unconstrained and equality constrained optimization. Institute of Computer Science, Academy of Sciences of the Czech Republic. Technical Report 767 (1999) |
| [36] | Marquardt, DW, An algorithm for least-squares estimation of nonlinear parameters, J. Soc. Ind. Appl. Math., 11, 431-441, 1963 · Zbl 0112.10505 |
| [37] | Narushima, Y.; Yabe, H.; Ford, JA, A three-term conjugate gradient method with sufficient descent property for unconstrained optimization, SIAM J. Optim., 21, 212-230, 2011 · Zbl 1250.90087 |
| [38] | Nocedal, J., Wright, S.J.: Numerical optimization. Springer series in operation Research, Springer, New York (1999) · Zbl 0930.65067 |
| [39] | Pola, E.; Ribiere, G., Note sur la convergence de methodes de directions conjugées, Rev. Française Informat. Rech. Oper., 16, 35-43, 1969 · Zbl 0174.48001 |
| [40] | Polyak, BT, The conjugate gradient method in extremal problems, USSR Comput. Math. Math. Phys., 9, 94-112, 1969 · Zbl 0229.49023 |
| [41] | Yabe, H.; Yamaki, N., Local and superlinear convergence of structured quasi-Newton methods for nonlinear optimization, J. Oper. Res. Soc. Japan, 39, 541-557, 1996 · Zbl 0874.90181 |
| [42] | Yu, G.; Guan, L.; Chen, W., Spectral conjugate gradient methods with sufficient descent property for large-scale unconstrained optimization, Optim. Methods Softw., 23, 275-293, 2008 · Zbl 1279.90166 |
| [43] | Yuan, G.; Li, T.; Hu, W., A conjugate gradient algorithm for large-scale nonlinear equations and image restoration problems, Appl. Numer. Math., 147, 129-141, 2020 · Zbl 1433.90165 |
| [44] | Yuan, G.; Lu, J.; Wang, Z., The modified PRP conjugate gradient algorithm under a non-descent line search and its application in the Muskingum model and image restoration problems, Soft Comput., 25, 5867-5879, 2021 · Zbl 1498.90176 |
| [45] | Zhang, JZ; Deng, NY; Chen, L., New quasi-Newton equation and related methods for unconstrained optimization, J. Optim. Theory Appl., 102, 147-167, 1999 · Zbl 0991.90135 |
| [46] | Zhang, JZ; Xue, Y.; Zhang, K., A structured secant method based on a new quasi-Newton equation for nonlinear least squares problems, BIT Numer. Math., 43, 217-229, 2003 · Zbl 1022.65070 |
| [47] | Zhang, L.; Zhou, W.; Li, D., Global convergence of a modified Fletcher-Reeves conjugate gradient method with Armijo-type line search, Numer. Math., 104, 561-572, 2006 · Zbl 1103.65074 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
