×

Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization. (English) Zbl 1318.90075

Authors’ abstract: We propose a first-order interior point algorithm for a class of non-Lipschitz and nonconvex minimization problems with box constraints, which arise from applications in variable selection and regularized optimization. The objective functions of these problems are continuously differentiable typically at interior points of the feasible set. Our first-order algorithm is easy to implement and the objective function value is reduced monotonically along the iteration points. We show that the worst-case iteration complexity for finding an \(\epsilon\)-scaled first-order stationary point is \(O(\epsilon^{-2})\). Furthermore, we develop a second-order interior point algorithm using the Hessian matrix, and solve a quadratic program with a ball constraint at each iteration. Although the second-order interior point algorithm costs more computational time than that of the first-order algorithm in each iteration, its worst-case iteration complexity for finding an \(\epsilon\)-scaled second-order stationary point is reduced to \(O(\epsilon^{-3/2})\). Note that an \(\epsilon\)-scaled second-order stationary point must also be an \(\epsilon\)-scaled first-order stationary point.
Reviewer: Do Van Luu (Hanoi)

MSC:

90C51 Interior-point methods
90C30 Nonlinear programming
90C26 Nonconvex programming, global optimization
49M37 Numerical methods based on nonlinear programming
65K05 Numerical mathematical programming methods

Software:

ElemStatLearn
Full Text: DOI

References:

[1] Bian, W., Chen, X.: Worst-case complexity of smoothing quadratic regularization methods for non-Lipschitzian optimization. SIAM J. Optim. 28, 1718-1741 (2013) · Zbl 1282.90175 · doi:10.1137/120864908
[2] Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51, 34-81 (2009) · Zbl 1178.68619 · doi:10.1137/060657704
[3] Cartis, C., Gould, N.I.M., Toint, PhL: On the evaluation complexity of composite function minimization with applications to nonconvex nonlinear programming. SIAM J. Optim. 21, 1721-1739 (2011) · Zbl 1236.90118 · doi:10.1137/11082381X
[4] Cartis, C., Gould, N.I.M., Toint, PhL: Adaptive cubic regularisation methods for unconstrained optimization, Part I: motivation, convergence and numerical results. Math. Program. 127, 245-295 (2011) · Zbl 1229.90192 · doi:10.1007/s10107-009-0286-5
[5] Cartis, C., Gould, N.I.M., Toint, PhL: Adaptive cubic regularisation methods for unconstrained optimization, Part II: worst-case function- and derivative-evaluation complexity. Math. Program. 130, 295-319 (2011) · Zbl 1229.90193 · doi:10.1007/s10107-009-0337-y
[6] Cartis, C., Gould, N.I.M., Toint, PhL: An adaptive cubic regularization algorithm for nonconvex optimization with convex constraints and its function-evaluation complexity. IMA J. Numer. Anal. 32, 1661-1695 (2012) · Zbl 1267.65061 · doi:10.1093/imanum/drr035
[7] Cartis, C., Gould, N.I.M., Toint, PhL: Complexity bounds for second-order optimality in unconstrained optimization. J. Complex. 28, 93-108 (2012) · Zbl 1245.65063 · doi:10.1016/j.jco.2011.06.001
[8] Chen, X.: Smoothing methods for nonsmooth, novonvex minimization. Math. Program. 134, 71-99 (2012) · Zbl 1266.90145 · doi:10.1007/s10107-012-0569-0
[9] Chen, X., Ge, D., Wang, Z., Ye, Y.: Complexity of unconstrained \[L_2\] L \[2-L_p\] Lp minimization. Math. Program. 143, 371-383 (2014) · Zbl 1285.90039
[10] Chen, X., Ng, M., Zhang, C.: Nonconvex \[l_p\] lp regularization and box constrained model for image restoration. IEEE Trans. Image Process. 21, 4709-4721 (2012) · Zbl 1373.94080 · doi:10.1109/TIP.2012.2214051
[11] Chen, X., Niu, L., Yuan, Y.: Optimality conditions and smoothing trust region Newton method for non-Lipschitz optimization. SIAM J. Optim. 23, 1528-1552 (2013) · Zbl 1291.90238 · doi:10.1137/120871390
[12] Chen, X., Xu, F., Ye, Y.: Lower bound theory of nonzero entries in solutions of \[l_2\] l \[2-l_p\] lp minimization. SIAM J. Sci. Comput. 32, 2832-2852 (2010) · Zbl 1242.90174 · doi:10.1137/090761471
[13] Chen, X., Zhou, W.: Smoothing nonlinear conjugate gradient method for image restoration using nonsmooth nonconvex minimization. SIAM J. Imaging Sci. 3, 765-790 (2010) · Zbl 1200.65031 · doi:10.1137/080740167
[14] Fan, J.: Comments on ’Wavelets in stastics: a review’ by A. Antoniadis. J. Ital. Stat. Soc. 6, 131-138 (1997) · Zbl 1454.62116 · doi:10.1007/BF03178906
[15] Garmanjani, R., Vicente, L.N.: Smoothing and worst case complexity for direct-search methods in nonsmooth optimization. IMA J. Numer. Anal. 33, 1008-1028 (2013) · Zbl 1272.65050
[16] Ge, D., Jiang, X., Ye, Y.: A note on the complexity of \[L_p\] Lp minimization. Math. Program. 21, 1721-1739 (2011) · Zbl 1236.90118
[17] Gratton, S., Sartenaer, A., Toint, PhL: Recursive trust-region methods for multiscale nonlinear optimization. SIAM J. Optim. 19, 414-444 (2008) · Zbl 1163.90024 · doi:10.1137/050623012
[18] Griewank, A.: The Modification of Newton’s Method for Unconstrained Optimization by Bounding Cubic Terms. Technical report NA/12 (1981). University of Cambridge, UK Department of Applied Mathematics and Theoretical Physics (1981) · Zbl 0894.90117
[19] Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning Data Mining, Inference, and Prediction. Springer, New York (2009) · Zbl 1273.62005
[20] Huang, J., Horowitz, J.L., Ma, S.: Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Stat. 36, 587-613 (2008) · Zbl 1133.62048 · doi:10.1214/009053607000000875
[21] Nesterov, Y.: Introductory Lectures on Convex Optimization, Applied Optimization. Kluwer, Dordrecht (2004) · Zbl 1086.90045 · doi:10.1007/978-1-4419-8853-9
[22] Nesterov, Y., Polyak, B.T.: Cubic regularization of Newton’s method and its global performance. Math. Program. 108, 177-205 (2006) · Zbl 1142.90500 · doi:10.1007/s10107-006-0706-8
[23] Nesterov, Y.: Accelerating the cubic regularization of Newton’s method on convex problems. Math. Program. 112, 159-181 (2008) · Zbl 1167.90013 · doi:10.1007/s10107-006-0089-x
[24] Nikolova, M., Ng, M.K., Zhang, S., Ching, W.-K.: Efficient reconstruction of piecewise constant images using nonsmooth nonconvex minimization. SIAM J. Imaging Sci. 1, 2-25 (2008) · Zbl 1207.94017 · doi:10.1137/070692285
[25] Tibshirani, R.: Shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B 58, 267-288 (1996) · Zbl 0850.62538
[26] Vavasis, S.A., Zippel, R.: Proving Polynomial Time for Sphere-Constrained Quadratic Programming, Technical Report 90-1182, Department of Computer Science. Cornell University, Ithaca, NY (1990) · Zbl 1236.90118
[27] Vavasis, S.A.: Nonlinear Optimization: Complexity Issues. Oxford Sciences, New York (1991) · Zbl 0785.90091
[28] Ye, Y.: Interior Point Algorithms: Theory and Analysis. Wiley, New York (1997) · Zbl 0943.90070 · doi:10.1002/9781118032701
[29] Ye, Y.: On the complexity of approximating a KKT point of quadratic programming. Math. Program. 80, 195-211 (1998) · Zbl 0894.90117 · doi:10.1007/BF01581726
[30] Ye, Y.; Floudas, C. (ed.); Pardalos, PM (ed.), A new complexity result on minimization of a quadratic function with a sphere constraint (1992), Princeton
[31] Zhang, C.-H.: Nearly unbiased variable selection under minimax concave penalty. Ann. Stat. 38, 894-942 (2010) · Zbl 1183.62120
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.