×

New computational guarantees for solving convex optimization problems with first order methods, via a function growth condition measure. (English) Zbl 1403.90549

Summary: Motivated by recent work of J. Renegar [“A framework for applying subgradient methods to conic optimization problems”, arXiv:1503.02611] we present new computational methods and associated computational guarantees for solving convex optimization problems using first-order methods. Our problem of interest is the general convex optimization problem \(f^* = \min _{x \in Q} f(x)\), where we presume knowledge of a strict lower bound \(f_{\mathrm{slb}}< f^*\). [Indeed, \(f_{\mathrm{slb}}\) is naturally known when optimizing many loss functions in statistics and machine learning (least-squares, logistic loss, exponential loss, total variation loss, etc.) as well as in Renegar’s transformed version of the standard conic optimization problem [loc. cit.]; in all these cases one has \(f_{\mathrm{slb}}= 0 < f^*\).] We introduce a new functional measure called the growth constant \(G\) for \(f(\cdot )\), that measures how quickly the level sets of \(f(\cdot )\) grow relative to the function value, and that plays a fundamental role in the complexity analysis. When \(f(\cdot )\) is non-smooth, we present new computational guarantees for the subgradient descent method and for smoothing methods, that can improve existing computational guarantees in several ways, most notably when the initial iterate \(x^0\) is far from the optimal solution set. When \(f(\cdot )\) is smooth, we present a scheme for periodically restarting the accelerated gradient method that can also improve existing computational guarantees when \(x^0\) is far from the optimal solution set, and in the presence of added structure we present a scheme using parametrically increased smoothing that further improves the associated computational guarantees.

MSC:

90C25 Convex programming
65K05 Numerical mathematical programming methods
68Q25 Analysis of algorithms and problem complexity
90C06 Large-scale problems in mathematical programming

References:

[1] Beck, A; Teboulle, M, Smoothing and first order methods: a unified framework, SIAM J. Optim., 22, 557-580, (2012) · Zbl 1251.90304 · doi:10.1137/100818327
[2] Bonnans, J., Ioffe, A.: Quadratic growth and stability in convex programming problems with multiple solutions. Technical report, INRIA Research Report RR-2403 (1994) · Zbl 0839.90090
[3] Burke, J; Deng, S, Weak sharp minima revisited, part I: basic theory, Control Cybern., 31, 439-469, (2002) · Zbl 1105.90356
[4] Burke, J; Deng, S, Weak sharp minima revisited, part II: application to linear regularity and error bounds, Math. Program., 104, 235-261, (2005) · Zbl 1124.90349 · doi:10.1007/s10107-005-0615-2
[5] Burke, J; Deng, S, Weak sharp minima revisited, part III: error bounds for differentiable convex inclusions, Math. Program., 116, 37-56, (2009) · Zbl 1163.90016 · doi:10.1007/s10107-007-0130-8
[6] Burke, J; Ferris, M, Weak sharp minima in mathematical programming, SIAM J. Control Optim., 31, 1340-1359, (1993) · Zbl 0791.90040 · doi:10.1137/0331063
[7] Burke, J; Ferris, M, A Gauss-Newton method for convex composite optimization, Math. Program., 71, 179-194, (1995) · Zbl 0846.90083 · doi:10.1007/BF01585997
[8] Burke, J; Lewis, A; Overton, M, Optimal stability and eigenvalue multiplicity, Found. Comput. Math., 1, 205-225, (2001) · Zbl 0994.15022 · doi:10.1007/PL00021726
[9] Ferris, M, Finite termination of the proximal point algorithm, Math. Program., 50, 359-366, (1991) · Zbl 0741.90051 · doi:10.1007/BF01594944
[10] Gilpin, A; Peña, J; Sandholm, T, First-order algorithm with O\((\ln (1/ϵ ))\) convergence for \(ϵ \)-equilibrium in two-person zero-sum games, Math. Program., 133, 279-298, (2012) · Zbl 1243.91004 · doi:10.1007/s10107-010-0430-2
[11] Hastie, T., Tibshirani, R., Friedman, J.: Elements of Statistical Learning. Springer Series in Statistics, 2nd edn. Springer, New York (2009) · Zbl 1273.62005
[12] Jourani, A, Hoffman’s error bound, local controllability, and sensitivity analysis, SIAM J. Control Optim., 38, 947-970, (2000) · Zbl 0945.46023 · doi:10.1137/S0363012998339216
[13] Nemirovsky, A., Yudin, D.: Problem Complexity and Method Efficiency in Optimization. Wiley, New York (1983) · Zbl 1126.90058
[14] Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, Boston (2003) · Zbl 1086.90045
[15] Nesterov, Y, Smooth minimization of non-smooth functions, Math. Program., 103, 127-152, (2005) · Zbl 1079.90102 · doi:10.1007/s10107-004-0552-5
[16] Nesterov, Y, Smoothing technique and its applications in semidefinite optimization, Math. Program., 110, 245-259, (2007) · Zbl 1126.90058 · doi:10.1007/s10107-006-0001-8
[17] O’Donoghue, B; Candes, E, Adaptive restart for accelerated gradient schemes, Found. Comput. Math., 15, 715-732, (2013) · Zbl 1320.90061 · doi:10.1007/s10208-013-9150-3
[18] Polyak, B.: Sharp minima. In: Proceedings of the IIASA Workshop on Generalized Lagrangians and Their Applications, Laxenburg, Austria. Institute of Control Sciences Lecture Notes, Moscow (1979)
[19] Polyak, B.: Introduction to Optimization. Optimization Software Inc., New York (1987) · Zbl 0708.90083
[20] Renegar, J.: Efficient first-order methods for linear programming and semidefinite programming. preprint arXiv:1409.5832 (2014) · Zbl 0741.90051
[21] Renegar, J.: Accelerated first-order methods for hyperbolic programming. Preprint arXiv:1512.07569 (2015) · Zbl 0994.15022
[22] Renegar, J.: A framework for applying subgradient methods to conic optimization problems (version 2). preprint arXiv:1503.02611v2 (2015) · Zbl 0846.90083
[23] Renegar, J, “efficient” subgradient methods for general convex optimization, SIAM J. Optim., 26, 2649-2676, (2016) · Zbl 1351.90129 · doi:10.1137/15M1027371
[24] Shapiro, A, Perturbation theory of nonlinear programs when the set of optimal solutions is not a singleton, Appl. Math. Optim., 18, 215-229, (1988) · Zbl 0654.90083 · doi:10.1007/BF01443623
[25] Shapiro, A, Perturbation analysis of optimization problems in Banach spaces, Numer. Funct. Anal. Optim., 13, 97-116, (1992) · Zbl 0763.49009 · doi:10.1080/01630569208816463
[26] Su, W., Boyd, S., Candes, E.: A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights. Adv. Neural Inf. Process. Syst. 2510-2518, (2014) · Zbl 1391.90667
[27] Tseng, P.: On accelerated proximal gradient methods for convex-concave optimization. Technical report, May 21 (2008)
[28] Yang, T., Lin, Q.: RSG: beating subgradient methods without smoothness and strong convexity. Preprint arXiv:1512.03107v12 (2015) · Zbl 1251.90304
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.