×

Conic optimization-based algorithms for nonnegative matrix factorization. (English) Zbl 1528.90181

Summary: Nonnegative matrix factorization is the following problem: given a nonnegative input matrix \(V\) and a factorization rank \(K\), compute two nonnegative matrices, \(W\) with \(K\) columns and \(H\) with \(K\) rows, such that WH approximates \(V\) as well as possible. In this paper, we propose two new approaches for computing high-quality NMF solutions using conic optimization. These approaches rely on the same two steps. First, we reformulate NMF as minimizing a concave function over a product of convex cones – one approach is based on the exponential cone and the other on the second-order cone. Then, we solve these reformulations iteratively: at each step, we minimize exactly, over the feasible set, a majorization of the objective functions obtained via linearization at the current iterate. Hence these subproblems are convex conic programs and can be solved efficiently using dedicated algorithms. We prove that our approaches reach a stationary point with an accuracy decreasing as \(\mathcal{O}\left( \frac{1}{i} \right)\), where \(i\) denotes the iteration number. To the best of our knowledge, our analysis is the first to provide a convergence rate to stationary points for NMF. Furthermore, in the particular cases of rank-1 factorizations (i.e. \(K = 1)\), we show that one of our formulations can be expressed as a convex optimization problem, implying that optimal rank-1 approximations can be computed efficiently. Finally, we show on several numerical examples that our approaches are able to frequently compute exact NMFs (i.e. with \(V = WH)\) and compete favourably with the state of the art.

MSC:

90C22 Semidefinite programming

Software:

Mosek

References:

[1] Abbaszadehpeivasti, H.; de Klerk, E.; Zamani, M., On the Rate of Convergence of the Difference-of-Convex Algorithm (DCA), J. Optim. Theory Appl. (2023) · Zbl 07904937 · doi:10.1007/s10957-023-02199-z
[2] Arora, S.; Ge, R.; Kannan, R.; Moitra, A., Computing a nonnegative matrix factorization—provably, SIAM J. Comput., 45, 1582-1611 (2016) · Zbl 1350.68123
[3] Basu, S.; Pollack, R.; Roy, M. F., On the combinatorial and algebraic complexity of quantifier elimination, J. ACM., 43, 1002-1045 (1996) · Zbl 0885.68070
[4] Berman, A.; Shaked-Monderer, N., Completely Positive Matrices (2003), World Scientific · Zbl 1030.15022
[5] Cichocki, A.; Zdunek, R.; Phan, A. H.; Amari, S.i., Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation (2009), John Wiley & Sons
[6] Clarkson, K. L., Coresets, sparse greedy approximation, and the Frank-Wolfe algorithm, ACM Trans. Algorithms., 6, 1-30 (2010) · Zbl 1300.90026
[7] Cohen, J. E.; Rothblum, U. G., Nonnegative ranks, decompositions, and factorizations of nonnegative matrices, Linear. Algebra. Appl., 190, 149-168 (1993) · Zbl 0784.15001
[8] Dewez, J., Computational approaches for lower bounds on the nonnegative rank, Ph.D. diss., UCLouvain, 2022. · Zbl 1447.35053
[9] Dewez, J.; Gillis, N.; Glineur, F., A geometric lower bound on the extension complexity of polytopes based on the f-vector, Discrete Appl Math., 303, 22-38 (2021) · Zbl 1477.52016
[10] Frank, M.; Wolfe, P., An algorithm for quadratic programming, Nav. Res. Logist. Q., 3, 95-110 (1956)
[11] Fu, X.; Huang, K.; Sidiropoulos, N. D.; Ma, W. K., Nonnegative matrix factorization for signal and data analytics: identifiability, algorithms, and applications, IEEE Signal Process. Mag., 36, 59-80 (2019)
[12] Gillis, N., Approximation et sous-approximation de matrices par factorisation positive : algorithmes, complexité et applications, Master’s thesis, Universite Catholique de Louvain, 2007.
[13] Gillis, N., Nonnegative Matrix Factorization (2020), SIAM: SIAM, Philadelphia
[14] Gillis, N.; Glineur, F., Accelerated multiplicative updates and hierarchical ALS algorithms for nonnegative matrix factorization, Neural. Comput., 24, 1085-1105 (2012)
[15] Gillis, N., Nonnegative matrix factorization: Complexity, algorithms and applications. Unpublished doctoral dissertation, Université catholique de Louvain. Louvain-La-Neuve: CORE. 2011.
[16] Gillis, N.; Glineur, F., Using underapproximations for sparse nonnegative matrix factorization, Pattern. Recognit., 43, 1676-1687 (2010) · Zbl 1191.68783
[17] Jaggi, M., Sparse convex optimization methods for machine learning, Ph.D. diss., ETH Zurich, 2011.
[18] Jaggi, M., Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization, in Proceedings of the 30th International Conference on Machine Learning, Proceedings of Machine Learning Research Vol. 28, 17-19 Jun, Atlanta, Georgia, USA. PMLR, 2013, pp. 427-435.
[19] Kim, J.; Park, H., Fast nonnegative matrix factorization: an active-set-like method and comparisons, SIAM J. Sci. Comput., 33, 3261-3281 (2011) · Zbl 1232.65068
[20] Krone, R.; Kubjas, K., Uniqueness of nonnegative matrix factorizations by rigidity theory, SIAM J. Matrix. Anal. Appl., 42, 134-164 (2021) · Zbl 1458.15028
[21] Kuang, D., Ding, C., and Park, H., Symmetric Nonnegative Matrix Factorization for Graph Clustering, in Proceedings of the 2012 SIAM International Conference on Data Mining. pp. 106-117.
[22] Lacoste-Julien, S., Convergence rate of Frank-Wolfe for non-convex objectives, Preprint (2016). Available at arXiv, 1607.00345.
[23] Le Thi, H. A.; Pham Dinh, T., DC programming and DCA: thirty years of developments, Math. Program., 169, 5-68 (2018) · Zbl 1387.90197
[24] Moitra, A., An Almost Optimal Algorithm for Computing Nonnegative Rank, in Proc. of the 24th Annual ACM-SIAM Symp. on Discrete Algorithms (SODA ’13), 2013, pp. 1454-1464. · Zbl 1422.68130
[25] Mosek APS, Optimization software. available at https://www.mosek.com/.
[26] Sharma, R.; Sharma, K., A new dual based procedure for the transportation problem, Eur. J. Oper. Res., 122, 611-624 (2000) · Zbl 0961.90065
[27] Shitov, Y., The nonnegative rank of a matrix: hard problems, easy solutions, SIAM Rev., 59, 794-800 (2017) · Zbl 1377.15003
[28] Tepper, M. and Sapiro, G., Nonnegative matrix underapproximation for robust multiple model fitting, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2059-2067.
[29] Vandaele, A.; Gillis, N.; Glineur, F.; Tuyttens, D., Heuristics for exact nonnegative matrix factorization, J. Glob. Optim., 65, 369-400 (2016) · Zbl 1341.65057
[30] Vavasis, S. A., On the complexity of nonnegative matrix factorization, SIAM J. Optim., 20, 1364-1377 (2010) · Zbl 1206.65130
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.