×

Convergence of inertial dynamics driven by sums of potential and nonpotential operators with implicit Newton-like damping. (English) Zbl 1531.37081

This paper presents an analysis of an implicit Newton-type inertial method. The aim of this method is to find \(x\in \mathcal{H}\) such that \( \nabla f(x) + B(x)=0 \), where \(\mathcal{H}\) is a real Hilbert space, \(f\) is a continuously differentiable convex function and \(B\) a monotone and cocoercive operator.
The authors consider a dynamical system whose stationary points are solutions to the following problem: \[ \ddot{x}(t) + \gamma \dot{x}(t) +\partial f(x(t) +\beta_f \dot{x}(t)) +B(x(t)+\beta_b \dot{x}(t)) = 0. \] The terms \(\partial f(x(t) +\beta_f \dot{x}(t))\) and \(B(x(t)+\beta_b \dot{x}(t))\) correspond to first-order Taylor expansions, which relates this scheme to a Newtown-type scheme.
The authors show that the considered dynamical system is well behaved, and that, under reasonable assumptions, the system has a unique strong global solution for any given initial condition (Cauchy data) \((x(0),\dot{x}(0))\). Furthermore, they show that every solution trajectory of the dynamical system asymptotically converges to a solution of the original problem \( \nabla f(x) + B(x)=0 \).
In the second part of the paper, a splitting proximal algorithm is derived, from the discretisation of the system and it is shown that under some conditions the algorithm produces a sequence converging (weakly) to a solution. Several variations of this algorithm are considered, such as perturbed systems or finite difference approaches.
The article is concluded with a numerical section showing on the basis of a simple example how the proposed scheme (called iDINAM) leads to attenuated oscillations in the convergence of the solution, as compared to the case when \(\beta_f=\beta_b=0\).

MSC:

37N40 Dynamical systems in optimization and economics
37M99 Approximation methods and numerical treatment of dynamical systems
49M15 Newton-type methods
49M25 Discrete approximations in optimal control
70F40 Problems involving a system of particles with friction
65K05 Numerical mathematical programming methods
65K10 Numerical optimization and variational techniques
Full Text: DOI

References:

[1] Abbas, B.; Attouch, H., Dynamical systems and forward-backward algorithms associated with the sum of a convex subdifferential and a monotone cocoercive operator, Optimization, 64, 10, 2223-2252 (2015) · Zbl 1345.34115 · doi:10.1080/02331934.2014.971412
[2] Abbas, B.; Attouch, H.; Svaiter, BF, Newton-like dynamics and forward-backward methods for structured monotone inclusions in Hilbert spaces, J. Optim. Theor. Appl., 161, 2, 331-360 (2014) · Zbl 1339.47080 · doi:10.1007/s10957-013-0414-5
[3] Adly, S.; Attouch, H., Finite convergence of proximal-gradient inertial algorithms combining dry friction with Hessian-driven damping, SIAM J. Optim., 30, 3, 2134-2162 (2020) · Zbl 1452.65103 · doi:10.1137/19M1307779
[4] Adly, S.; Attouch, H.; Vo, VN, Asymptotic behavior of Newton-like inertial dynamics involving the sum of potential and nonpotential terms, Fixed Point Theor. Algorithms Sci. Eng., 17, 30 (2021) · Zbl 07525621
[5] Adly, S.; Attouch, H.; Vo, VN, Newton-type inertial algorithms for solving monotone equations governed by sums of potential and nonpotential operators, Appl. Math. Optim., 85, 3, 31 (2022) · Zbl 1498.65072 · doi:10.1007/s00245-022-09846-3
[6] Alecsa, CD; László, S.; Pinta, T., An extension of the second order dynamical system that models Nesterov’s convex gradient method, Appl. Math. Optim., 84, 2, 1687-1716 (2021) · Zbl 1486.34050 · doi:10.1007/s00245-020-09692-1
[7] Alvarez, F., On the minimizing property of a second order dissipative system in Hilbert space, SIAM J. Control Optim., 38, 4, 1102-1119 (2000) · Zbl 0954.34053 · doi:10.1137/S0363012998335802
[8] Apidopoulos, V.; Aujol, J-F; Dossal, Ch, The differential inclusion modeling FISTA algorithm and optimality of convergence rate in the case \(b\le 3\), SIAM J. Optim., 28, 551-574 (2018) · Zbl 1408.34024 · doi:10.1137/17M1128642
[9] Alvarez, F.; Attouch, H., An inertial proximal method for maximal monotone operators via discretization of a nonlinear oscillator with damping, Set Valued Anal., 9, 1-2, 3-11 (2001) · Zbl 0991.65056 · doi:10.1023/A:1011253113155
[10] Attouch H., Boţ R.I., Nguyen D.-K., Fast convex optimization via time scale and averaging of the steepest descent, arXiv:2208.08260v1 [math.OC] Aug (2022)
[11] Alvarez, F.; Attouch, H.; Bolte, J.; Redont, P., A second-order gradient-like dissipative dynamical system with Hessian-driven damping: application to optimization and mechanics, J. Math. Pures Appl., 81, 8, 747-779 (2002) · Zbl 1036.34072 · doi:10.1016/S0021-7824(01)01253-3
[12] Attouch, H.; Cabot, A., Convergence rates of inertial forward-backward algorithms, SIAM J. Optim., 28, 1, 849-874 (2018) · Zbl 1387.49047 · doi:10.1137/17M1114739
[13] Attouch, H.; Chbani, Z.; Peypouquet, J.; Redont, P., Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity, Math. Program. Ser. B, 168, 123-175 (2018) · Zbl 1395.34068 · doi:10.1007/s10107-016-0992-8
[14] Attouch, H.; Chbani, Z.; Riahi, H., Rate of convergence of the Nesterov accelerated gradient method in the subcritical case \(\alpha \le 3\), ESAIM Control Optim. Calc. Var., 25, 2, 34 (2019) · Zbl 1437.49045
[15] Attouch, H.; Chbani, Z.; Fadili, J.; Riahi, H., First-order algorithms via inertial systems with Hessian driven damping, Math. Program. Ser. A, 193, 1, 113-155 (2022) · Zbl 1497.37121 · doi:10.1007/s10107-020-01591-1
[16] Attouch, H.; Chbani, Z.; Fadili, J.; Riahi, H., Convergence of iterates for first-order optimization algorithms with inertia and Hessian driven damping, Optimization (2021) · Zbl 1519.90260 · doi:10.1080/02331934.2021.2009828
[17] Attouch, H.; Chbani, Z.; Fadili, J.; Riahi, H., Fast convergence of dynamical ADMM via time scaling of damped inertial dynamics, J. Optim. Theor. Appl., 193, 704-736 (2022) · Zbl 1497.37122 · doi:10.1007/s10957-021-01859-2
[18] Attouch, H., Fadili, J., Kungurtsev, V.: On the effect of perturbations in first-order optimization methods with inertia and Hessian driven damping. Evolution Equations and Control Theory. 12(1), (2022) · Zbl 1509.37132
[19] Attouch, H.; László, SC, Continuous Newton-like inertial dynamics for monotone inclusions, Set Valued Var. Anal., 29, 3, 555-581 (2021) · Zbl 1477.37101 · doi:10.1007/s11228-020-00564-y
[20] Attouch, H.; László, SC, Newton-like inertial dynamics and proximal algorithms governed by maximally monotone operators, SIAM J. Optim., 30, 4, 3252-3283 (2020) · Zbl 1477.90058 · doi:10.1137/20M1333316
[21] Attouch, H.; Maingé, PE, Asymptotic behavior of second order dissipative evolution equations combining potential with nonpotential effects, ESAIM Control Optim. Calc. Var., 17, 3, 836-857 (2011) · Zbl 1230.34051 · doi:10.1051/cocv/2010024
[22] Attouch, H.; Marques Alves, M.; Svaiter, BF, A dynamic approach to a proximal-Newton method for monotone inclusions in Hilbert Spaces, with complexity \(\cal{O} (1/n^2)\), J. Convex Anal., 23, 1, 139-180 (2016) · Zbl 1348.47053
[23] Attouch, H.; Peypouquet, J., Convergence of inertial dynamics and proximal algorithms governed by maximal monotone operators, Math. Program., 174, 1-2, 391-432 (2019) · Zbl 1412.37083 · doi:10.1007/s10107-018-1252-x
[24] Attouch, H.; Peypouquet, J.; Redont, P., Fast convex minimization via inertial dynamics with Hessian driven damping, J. Differ. Equ., 261, 10, 5734-5783 (2016) · Zbl 1375.49028 · doi:10.1016/j.jde.2016.08.020
[25] Attouch, H.; Redont, P.; Svaiter, BF, Global convergence of a closed-loop regularized Newton method for solving monotone inclusions in Hilbert spaces, J. Optim. Theor. Appl., 157, 3, 624-650 (2013) · Zbl 1290.90061 · doi:10.1007/s10957-012-0222-3
[26] Attouch, H.; Svaiter, BF, A continuous dynamical Newton-like approach to solving monotone inclusions, SIAM J. Control Optim., 49, 2, 574-598 (2011) · Zbl 1229.34097 · doi:10.1137/100784114
[27] Aujol, J.-F., Dossal, C., Hoàng, V., Labarrière, H., Rondepierre A.: Fast convergence of inertial dynamics with hessian-driven damping under geometry assumptions. arXiv:2206.06853, (2022)
[28] Baillon, J-B; Haddad, G., Quelques propriétés des opérateurs angles-bornés et n-cycliquement monotones, Israel J. Math., 26, 137-150 (1977) · Zbl 0352.47023 · doi:10.1007/BF03007664
[29] Bauschke, H.; Combettes, PL, Convex Analysis and Monotone Operator Theory in Hilbert spaces, CMS Books in Mathematics (2011), Springer · Zbl 1218.47001
[30] Beck, A.; Teboulle, M., A fast iterative shrinkage-thresholding algorithm for linear inverse problems, SIAM J. Imaging Sci., 2, 1, 183-202 (2009) · Zbl 1175.94009 · doi:10.1137/080716542
[31] Boţ, RI; Csetnek, ER, Second order forward-backward dynamical systems for monotone inclusion problems, SIAM J. Control Optim., 54, 1423-1443 (2016) · Zbl 1339.34070 · doi:10.1137/15M1012657
[32] Boţ, RI; Csetnek, ER; László, SC, Tikhonov regularization of a second order dynamical system with Hessian damping, Math. Program. Ser. B, 189, 1-2, 151-186 (2021) · Zbl 1489.34088 · doi:10.1007/s10107-020-01528-8
[33] Brézis, H.: Opérateurs maximaux monotones dans les espaces de Hilbert et équations d’évolution. Lecture Notes 5. North Holland (1972)
[34] Castera, C., Bolte, J., Févotte, C., Pauwels, E.: An Inertial Newton algorithm for deep learning. (2019), HAL-02140748 · Zbl 1482.68211
[35] Chambolle, A.; Dossal, Ch, On the convergence of the iterates of the fast iterative shrinkage thresholding algorithm, J. Optim. Theor. Appl., 166, 968-982 (2015) · Zbl 1371.65047 · doi:10.1007/s10957-015-0746-4
[36] Gelfand I.M. , Tsetlin M.: Printszip nelokalnogo poiska v sistemah avtomatich. Optimizatsii, Dokl. AN SSSR. 137, 295-298 (1961) (in Russian)
[37] Lin, T.; Jordan, MI, A control-theoretic perspective on optimal high-order optimization, Math. Program., 195, 929-975 (2022) · Zbl 1506.90205 · doi:10.1007/s10107-021-01721-3
[38] Muehlebach, M., Jordan, M. I.: A Dynamical systems perspective on nesterov acceleration. in Proceedings of the International Conference on Machine Learning, (2019)
[39] Nesterov, Y.: A method for solving the convex programming problem with convergence rate \(O(1/k^2)\). in Dokl. Akad. Nauk SSSR, (Russian) 269(3), 543-547 (1983)
[40] Polyak, BT, Some methods of speeding up the convergence of iteration methods, USSR Comput. Math. Math. Phys., 4, 1-17 (1964) · Zbl 0147.35301 · doi:10.1016/0041-5553(64)90137-5
[41] Peypouquet, J.; Sorin, S., Evolution equations for maximal monotone operators asymptotic analysis in continuous and discrete time, J. Convex Anal., 17, 3-4, 1113-1163 (2010) · Zbl 1214.47080
[42] Shi, B.; Du, SS; Jordan, MI; Su, WJ, Understanding the acceleration phenomenon via high-resolution differential equations, Math. Program., 195, 79-148 (2022) · Zbl 1500.65026 · doi:10.1007/s10107-021-01681-8
[43] Su, W.; Boyd, S.; Candès, EJ, A differential equation for modeling Nesterov’s accelerated gradient method, J. Mach. Learn. Res., 17, 153, 43 (2016) · Zbl 1391.90667
[44] Villa, S.; Salzo, S.; Baldassarres, L.; Verri, A., Accelerated and inexact forward-backward, SIAM J. Optim., 23, 3, 1607-1633 (2013) · Zbl 1295.90049 · doi:10.1137/110844805
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.