×

A gradient sampling method on algebraic varieties and application to nonsmooth low-rank optimization. (English) Zbl 1428.49015

Summary: In this paper, a nonsmooth optimization method for locally Lipschitz functions on real algebraic varieties is developed. To this end, the set-valued map \(\varepsilon\)-conditional subdifferential \(x\to\partial_{\varepsilon}^Nf(x):=\partial_{\varepsilon}f(x)+N(x)\) is introduced, where \(\partial_{\varepsilon}f(x)\) is the Goldstein-\(\varepsilon\)-subdifferential and \(N(x)\) is a closed convex cone at \(x\). It is proved that negative of the shortest \(\varepsilon\)-conditional subgradient provides a descent direction in \(T(x)\), which denotes the polar of \(N(x)\). The \(\varepsilon\)-conditional subdifferential at an iterate \(x_{\ell}\) can be approximated by a convex hull of a finite set of projected gradients at sampling points in \(x_\ell+\varepsilon_{\ell} B_{T(x_{\ell})}(0,1)\) to \(T(x_{\ell})\), where \(T(x_{\ell})\) is a linear space in the Bouligand tangent cone and \(B_{T(x_{\ell})}(0,1)\) denotes the unit ball in \(T(x_{\ell})\). The negative of the shortest vector in this convex hull is shown to be a descent direction in the Bouligand tangent cone at \(x_{\ell}\). The proposed algorithm makes a step along this descent direction with a certain step-size rule, followed by a retraction to lift back to points on the algebraic variety \(\mathcal{M}\). The convergence of the resulting algorithm to a critical point is proved. For numerical illustration, the considered method is applied to some nonsmooth problems on varieties of low-rank matrices \(\mathcal{M}_{\leq r}\) of real \(M\times N\) matrices of rank at most \(r\), specifically robust low-rank matrix approximation and recovery in the presence of outliers.

MSC:

49J52 Nonsmooth analysis
65K05 Numerical mathematical programming methods
14P05 Real algebraic sets
15A99 Basic linear algebra
49J53 Set-valued and variational analysis

Software:

GradSamp; Manopt
Full Text: DOI

References:

[1] P.-A. Absil, R. Mahony, and R. Sepulchre, Optimization Algorithms on Matrix Manifolds, Princeton University Press, Princeton, NJ, 2008. · Zbl 1147.65043
[2] A. Bagirov, N. Karmitsa, and M. M. Mäkelä, Introduction to Nonsmooth Optimization, Springer, Cham, 2014. · Zbl 1312.90053
[3] N. Boumal, B. Mishra, P.-A. Absil, and R. Sepulchre, Manopt, a MATLAB toolbox for optimization on manifolds, J. Mach. Learn. Res., 15 (2014), pp. 1455-1459, http://www.manopt.org. · Zbl 1319.90003
[4] J. V. Burke, A. S. Lewis, and M. L. Overton, A robust gradient sampling algorithm for nonsmooth, nonconvex optimization, SIAM J. Optim., 15 (2005), pp. 751-779. · Zbl 1078.65048
[5] L. Cambier and P.-A. Absil, Robust low-rank matrix completion by Riemannian optimization, SIAM J. Sci. Comput., 38 (2016), pp. S440-S460. · Zbl 1352.65149
[6] T. P. Cason, P.-A. Absil, and P. Van Dooren, Iterative methods for low rank approximation of graph similarity matrices, Linear Algebra Appl., 438 (2013), pp. 1863-1882. · Zbl 1264.65060
[7] F. H. Clarke, Optimization and Nonsmooth Analysis, 2nd ed., SIAM, Philadelphia, PA, 1990. · Zbl 0696.49002
[8] A. A. Goldstein, Optimization of Lipschitz continuous functions, Math. Program., 13 (1977), pp. 14-22. · Zbl 0394.90088
[9] P. Grohs and S. Hosseini, Nonsmooth trust region algorithms for locally Lipschitz functions on Riemannian manifolds, IMA J. Numer. Anal., 36 (2016), pp. 1167-1192. · Zbl 1433.90124
[10] P. Grohs and S. Hosseini, \( \varepsilon \)-subgradient algorithms for locally Lipschitz functions on Riemannian manifolds, Adv. Comput. Math., 42 (2016), pp. 333-360. · Zbl 1338.49029
[11] S. Hosseini, W. Huang, and R. Yousefpour, Line search algorithms for locally Lipschitz functions on Riemannian manifolds, SIAM J. Optim., 28 (2018), pp. 596-619. · Zbl 1387.49019
[12] S. Hosseini and A. Uschmajew, A Riemannian gradient sampling algorithm for nonsmooth optimization on manifolds, SIAM J. Optim., 27 (2017), pp. 173-189. · Zbl 1357.49062
[13] V. Y. Kaloshin, A geometric proof of the existence of Whitney stratifications, Mosc. Math. J., 5 (2005), pp. 125-133. · Zbl 1083.14067
[14] K. C. Kiwiel, Convergence of the gradient sampling algorithm for nonsmooth nonconvex optimization, SIAM J. Optim., 18 (2007), pp. 379-388. · Zbl 1149.65043
[15] D. Kressner, M. Steinlechner, and B. Vandereycken, Low-rank tensor completion by Riemannian optimization, BIT, 54 (2014), pp. 447-468. · Zbl 1300.65040
[16] D. B. O’Shea and L. C. Wilson, Limits of tangent spaces to real surfaces, Amer. J. Math., 126 (2004), pp. 951-980. · Zbl 1071.14058
[17] R. Schneider and A. Uschmajew, Convergence results for projected line-search methods on varieties of low-rank matrices via Łojasiewicz inequality, SIAM J. Optim., 25 (2015), pp. 622-646. · Zbl 1355.65079
[18] M. Tan, I. W. Tsang, L. Wang, B. Vandereycken, and S. J. Pan, Riemannian pursuit for big matrix recovery, in Proceedings of the 31st International Conference on Machine Learning (ICML 2014), 2014, pp. 1539-1547.
[19] A. Uschmajew and B. Vandereycken, Greedy rank updates combined with Riemannian descent methods for low-rank optimization, in Proceedings of the 2015 International Conference on Sampling Theory and Applications (SampTA), 2015, pp. 420-424.
[20] B. Vandereycken, Low-rank matrix completion by Riemannian optimization, SIAM J. Optim., 23 (2013), pp. 1214-1236. · Zbl 1277.15021
[21] B. Vandereycken and S. Vandewalle, A Riemannian optimization approach for computing low-rank solutions of Lyapunov equations, SIAM J. Matrix Anal. Appl., 31 (2010), pp. 2553-2579. · Zbl 1221.65108
[22] H. Whitney, Local properties of analytic varieties, in Differential and Combinatorial Topology (A Symposium in Honor of Marston Morse), Princeton University Press, Princeton, NJ 1965, pp. 205-244. · Zbl 0129.39402
[23] H. Whitney, Tangents to an analytic variety, Ann. of Math. (2), 81 (1965), pp. 496-549. · Zbl 0152.27701
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.