×

Mode-wise tensor decompositions: multi-dimensional generalizations of CUR decompositions. (English) Zbl 07415128

Summary: Low rank tensor approximation is a fundamental tool in modern machine learning and data science. In this paper, we study the characterization, perturbation analysis, and an efficient sampling strategy for two primary tensor CUR approximations, namely Chidori and Fiber CUR. We characterize exact tensor CUR decompositions for low multilinear rank tensors. We also present theoretical error bounds of the tensor CUR approximations when (adversarial or Gaussian) noise appears. Moreover, we show that low cost uniform sampling is sufficient for tensor CUR approximations if the tensor has an incoherent structure. Empirical performance evaluations, with both synthetic and real-world datasets, establish the speed advantage of the tensor CUR approximations over other state-of-the-art low multilinear rank tensor approximations.

MSC:

68T05 Learning and adaptive systems in artificial intelligence

References:

[1] Evrim Acar and B¨ulent Yener. Unsupervised multiway data analysis: A literature survey. IEEE Transactions on Knowledge and Data Engineering, 21(1):6-20, 2008.
[2] Salman Ahmadi-Asl, Stanislav Abukhovich, Maame G Asante-Mensah, Andrzej Cichocki, Anh Huy Phan, Tohishisa Tanaka, and Ivan Oseledets. Randomized algorithms for computation of Tucker decomposition and higher order SVD (HOSVD).IEEE Access, 9: 28684-28706, 2021.
[3] Akram Aldroubi, Ali Sekmen, Ahmet Bugra Koku, and Ahmet Faruk C¸ akmak. Similarity matrix framework for data from union of subspaces.Applied and Computational Harmonic Analysis, 45(2):425-435, 2018. · Zbl 1406.62066
[4] Akram Aldroubi, Keaton Hamm, Ahmet Bugra Koku, and Ali Sekmen. CUR decompositions, similarity matrices, and subspace clustering.Frontiers in Applied Mathematics and Statistics, 4:65, 2019.
[5] Casey Battaglino, Grey Ballard, and Tamara G Kolda. A practical randomized CP tensor decomposition.SIAM Journal on Matrix Analysis and Applications, 39(2):876-901, 2018. · Zbl 1444.65016
[6] Andrea L Bertozzi and Arjuna Flenner. Diffuse interface models on graphs for classification of high dimensional data.SIAM Review, 58(2):293-328, 2016. · Zbl 1339.68287
[7] Christos Boutsidis and David P Woodruff. Optimal CUR matrix decompositions.SIAM Journal on Computing, 46(2):543-589, 2017. · Zbl 1359.65059
[8] HanQin Cai, Jian-Feng Cai, and Ke Wei. Accelerated alternating projections for robust principal component analysis.The Journal of Machine Learning Research, 20(1):685- 717, 2019. · Zbl 1483.62098
[9] HanQin Cai, Keaton Hamm, Longxiu Huang, Jiaqi Li, and Tao Wang. Rapid robust principal component analysis: CUR accelerated inexact low rank estimation.IEEE Signal Processing Letters, 28:116-120, 2021a.
[10] HanQin Cai, Keaton Hamm, Longxiu Huang, and Deanna Needell. Robust CUR decomposition: Theory and imaging applications.arXiv preprint arXiv:2101.05231, 2021b.
[11] Cesar F Caiafa and Andrzej Cichocki. Generalizing the column-row matrix decomposition to multi-way arrays.Linear Algebra and its Applications, 433(3):557-573, 2010. · Zbl 1194.15025
[12] Emmanuel Candes and Justin Romberg. Sparsity and incoherence in compressive sampling. Inverse Problems, 23(3):969, 2007. · Zbl 1120.94005
[13] Maolin Che, Yimin Wei, and Hong Yan. Randomized algorithms for the low multilinear rank approximations of tensors.Journal of Computational and Applied Mathematics, 390: 113380, 2021. · Zbl 1462.65045
[14] Dehua Cheng, Richard Peng, Yan Liu, and Ioakeim Perros. SPALS: Fast alternating least squares via implicit leverage scores sampling.Advances in Neural Information Processing Systems, 29:721-729, 2016.
[15] Jiawei Chiu and Laurent Demanet. Sublinear randomized algorithms for skeleton decompositions.SIAM Journal on Matrix Analysis and Applications, 34(3):1361-1383, 2013. · Zbl 1277.68296
[16] Ali Civril and Malik Magdon-Ismail. On selecting a maximum volume sub-matrix of a matrix and related problems.Theoretical Computer Science, 410(47-49):4801-4811, 2009. · Zbl 1181.15002
[17] Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. A multilinear singular value decomposition.SIAM Journal on Matrix Analysis and Applications, 21(4):1253-1278, 2000a. · Zbl 0962.15005
[18] Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. On the best rank-1 and rank(r1, ..., rn) approximation of higher-order tensors.SIAM Journal on Matrix Analysis and Applications, 21(4):1324-1342, 2000b. · Zbl 0958.15026
[19] Petros Drineas, Ravi Kannan, and Michael W Mahoney. Fast Monte Carlo algorithms for matrices III: Computing a compressed approximate matrix decomposition.SIAM Journal on Computing, 36(1):184-206, 2006. · Zbl 1111.68149
[20] Petros Drineas, Michael W Mahoney, and S Muthukrishnan. Relative-error CUR matrix decompositions.SIAM Journal on Matrix Analysis and Applications, 30(2):844-881, 2008. · Zbl 1183.68738
[21] Bo Du, Mengfei Zhang, Lefei Zhang, Ruimin Hu, and Dacheng Tao. PLTD: Patch-based lowrank tensor decomposition for hyperspectral images.IEEE Transactions on Multimedia, 19(1):67-79, 2016.
[22] N Benjamin Erichson, Krithika Manohar, Steven L Brunton, and J Nathan Kutz. Randomized CP tensor decomposition.Machine Learning: Science and Technology, 1(2):025012, 2020.
[23] David H Foster, S´ergio MC Nascimento, and Kinjiro Amano. Information limits on neural identification of colored surfaces in natural scenes.Visual neuroscience, 21(3):331, 2004.
[24] Alan Frieze, Ravi Kannan, and Santosh Vempala. Fast Monte-Carlo algorithms for finding low-rank approximations.Journal of the ACM (JACM), 51(6):1025-1041, 2004. · Zbl 1125.65005
[25] Alex Gittens and Michael W Mahoney. Revisiting the Nystr¨om method for improved largescale machine learning.The Journal of Machine Learning Research, 17(1):3977-4041, 2016. · Zbl 1367.68223
[26] Alex Gittens, Kareem Aggour, and B¨ulent Yener. Adaptive sketching for fast and convergent Canonical Polyadic decomposition. InInternational Conference on Machine Learning, pages 3566-3575. PMLR, 2020.
[27] Sergei A Goreinov, Ivan V Oseledets, Dimitry V Savostyanov, Eugene E Tyrtyshnikov, and Nikolay L Zamarashkin. How to find a good submatrix. InMatrix Methods: Theory, Algorithms And Applications: Dedicated to the Memory of Gene Golub, pages 247-256. World Scientific, 2010. · Zbl 1215.65078
[28] Sergei A. Gore˘ınov, Nikolai L. Zamarashkin, and Eugene E. Tyrtyshnikov. Pseudo-skeleton approximations.Doklay Akdemii Nauk, 343(2):151-152, 1995.
[29] Sergei A Gore˘ınov, Nikolai Leonidovich Zamarashkin, and Evgenii Evgen’evich Tyrtyshnikov. Pseudo-skeleton approximations by matrices of maximal volume.Mathematical Notes, 62(4):515-519, 1997. · Zbl 0916.65040
[30] Lars Grasedyck. Hierarchical singular value decomposition of tensors.SIAM Journal on Matrix Analysis and Applications, 31(4):2029-2054, 2010. · Zbl 1210.65090
[31] Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2):217-288, 2011. · Zbl 1269.65043
[32] Keaton Hamm and Longxiu Huang. Stability of sampling for CUR decompositions.Foundations of Data Science, 2(2):83-99, 2020a.
[33] Keaton Hamm and Longxiu Huang. Perspectives on CUR decompositions.Applied and Computational Harmonic Analysis, 48(3):1088-1099, 2020b. · Zbl 1432.15014
[34] Keaton Hamm and Longxiu Huang. Perturbations of CUR decompositions.SIAM Journal on Matrix Analysis and Applications, 42(1):351-375, 2021. · Zbl 07331677
[35] Frank L Hitchcock. The expression of a tensor or a polyadic as a sum of products.Journal of Mathematical Physics, 6(1-4):164-189, 1927. · JFM 53.0095.01
[36] Frank L Hitchcock. Multiple invariants and generalized rank of ap-way matrix or tensor. Journal of Mathematical Physics, 7(1-4):39-79, 1928. · JFM 53.0097.01
[37] Misha E Kilmer, Karen Braman, Ning Hao, and Randy C Hoover. Third-order tensors as operators on matrices: A theoretical and computational framework with applications in imaging.SIAM Journal on Matrix Analysis and Applications, 34(1):148-172, 2013. · Zbl 1269.65044
[38] Tamara G Kolda and Brett W Bader. Tensor decompositions and applications.SIAM Review, 51(3):455-500, 2009. · Zbl 1173.65029
[39] Pieter M Kroonenberg and Jan De Leeuw. Principal component analysis of three-mode data by means of alternating least squares algorithms.Psychometrika, 45(1):69-97, 1980. · Zbl 0431.62035
[40] Joseph B Kruskal. Rank, decomposition, and uniqueness for 3-way and N-way arrays. Multiway Data Analysis, pages 7-18, 1989.
[41] Rui Li, Zhibin Pan, Yang Wang, and Ping Wang. The correlation-based Tucker decomposition for hyperspectral image compression.Neurocomputing, 419:357-370, 2021.
[42] Michael W Mahoney and Petros Drineas. CUR matrix decompositions for improved data analysis.Proceedings of the National Academy of Sciences, 106(3):697-702, 2009. · Zbl 1202.68480
[43] Michael W Mahoney, Mauro Maggioni, and Petros Drineas. Tensor-CUR decompositions for tensor-based data.SIAM Journal on Matrix Analysis and Applications, 30(3):957-987, 2008. · Zbl 1168.65340
[44] George Matsaglia and George PH Styan. Equalities and inequalities for ranks of matrices. Linear and multilinear Algebra, 2(3):269-292, 1974. · Zbl 0297.15003
[45] Aleksandr Mikhalev and Ivan V Oseledets. Rectangular maximum-volume submatrices and their applications.Linear Algebra and its Applications, 538:187-211, 2018. · Zbl 1374.15016
[46] Larsson Omberg, Joel R Meyerson, Kayta Kobayashi, Lucy S Drury, John FX Diffley, and Orly Alter. Global effects of DNA replication and DNA replication origin activity on eukaryotic gene expression.Molecular systems biology, 5(1):312, 2009.
[47] Ivan V Oseledets. Tensor-train decomposition.SIAM Journal on Scientific Computing, 33 (5):2295-2317, 2011. · Zbl 1232.15018
[48] R. Penrose. On best approximate solutions of linear matrix equations.Mathematical Proceedings of the Cambridge Philosophical Society, 52(1):17-19, 1956. · Zbl 0070.12501
[49] Anh-Huy Phan, Petr Tichavsk‘y, and Andrzej Cichocki. CANDECOMP/PARAFAC decomposition of high-order tensors through tensor reshaping.IEEE Transactions on Signal Processing, 61(19):4847-4860, 2013.
[50] Mark Rudelson and Roman Vershynin. Sampling from large matrices: An approach through geometric functional analysis.Journal of the ACM (JACM), 54(4):21-es, 2007. · Zbl 1326.68333
[51] Nicholas D Sidiropoulos, Lieven De Lathauwer, Xiao Fu, Kejun Huang, Evangelos E Papalexakis, and Christos Faloutsos. Tensor decomposition for signal processing and machine learning.IEEE Transactions on Signal Processing, 65(13):3551-3582, 2017. · Zbl 1415.94232
[52] Zhao Song, David P Woodruff, and Peilin Zhong. Relative error tensor low rank approximation. InProc. ACM-SIAM Symposium on Discrete Algorithms, pages 2772-2789. SIAM, 2019. · Zbl 1432.68584
[53] Danny C Sorensen and Mark Embree. A DEIM induced CUR factorization.SIAM Journal on Scientific Computing, 38(3):A1454-A1482, 2016. · Zbl 1382.65121
[54] Ameet Talwalkar and Afshin Rostamizadeh. Matrix coherence and the Nystr¨om method. InProceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, pages 572-579, 2010.
[55] Joel A Tropp. Improved analysis of the subsampled randomized Hadamard transform. Advances in Adaptive Data Analysis, 3(01n02):115-126, 2011. · Zbl 1232.15029
[56] Joel A Tropp. User-friendly tail bounds for sums of random matrices.Found. Comput. Math., 12(4):389-434, 2012. · Zbl 1259.60008
[57] Ledyard R Tucker. Some mathematical notes on three-mode factor analysis.Psychometrika, 31(3):279-311, 1966.
[58] Nick Vannieuwenhoven, Raf Vandebril, and Karl Meerbergen. A new truncation strategy for the higher-order singular value decomposition.SIAM Journal on Scientific Computing, 34(2):A1027-A1052, 2012. · Zbl 1247.65055
[59] M Alex O Vasilescu and Demetri Terzopoulos. Multilinear analysis of image ensembles: Tensorfaces. InEuropean conference on computer vision, pages 447-460, 2002. · Zbl 1034.68693
[60] Sergey Voronin and Per-Gunnar Martinsson. Efficient algorithms for CUR and interpolative matrix decompositions.Advances in Computational Mathematics, 43(3):495-516, 2017. · Zbl 1369.65049
[61] Lele Wang, Kun Xie, Thabo Semong, and Huibin Zhou. Missing data recovery based on tensor-CUR decomposition.IEEE Access, 6:532-544, 2017.
[62] Shusen Wang and Zhihua Zhang. Improving CUR matrix decomposition and the Nystr¨om approximation via adaptive sampling.The Journal of Machine Learning Research, 14(1): 2729-2769, 2013. · Zbl 1318.65023
[63] Christopher Williams and Matthias Seeger. Using the Nystr¨om method to speed up kernel machines. InProceedings of the 14th annual conference on neural information processing systems, pages 682-688, 2001.
[64] David Woodruff. Sketching as a tool for numerical linear algebra.Foundations and Trends in Theoretical Computer Science, 10(1-2):1-157, 2014. · Zbl 1316.65046
[65] Dong Xia, Ming Yuan, Cun-Hui Zhang, et al. Statistically optimal and computationally efficient low rank tensor completion from noisy entries.Annals of Statistics, 49(1):76-99, 2021. · Zbl 1473.62184
[66] Shuicheng Yan, Dong Xu, Qiang Yang, Lei Zhang, Xiaoou Tang, and Hong-Jiang Zhang. Multilinear discriminant analysis for face recognition.IEEE Transactions on Image Processing, 16(1):212-220, 2006.
[67] Jiyan Yang, Oliver Rubel, Michael W Mahoney, and Benjamin P Bowen. Identifying important ions and positions in mass spectrometry imaging data using CUR matrix decompositions.Analytical Chemistry, 87(9):4658-4666, 2015.
[68] B¨ulent Yener, Evrim Acar, Pheadra Aguis, Kristin Bennett, Scott L Vandenberg, and George E Plopper. Multiway modeling and analysis in stem cell systems biology.BMC Systems Biology, 2(1):63, 2008.
[69] Ching-Wa Yip, Michael W Mahoney, Alexander S Szalay, Istv´an Csabai, Tam´as Budav´ari, Rosemary FG Wyse, and Laszlo Dobos. Objective identification of informative wavelength regions in galaxy spectra.The Astronomical Journal, 147(5):110, 2014.
[70] Ali Zare, Alp Ozdemir, Mark A Iwen, and Selin Aviyente. Extension of PCA to higher order data structures: An introduction to tensors, tensor decompositions, and tensor PCA.Proceedings of the IEEE, 106(8):1341-1358, 2018.
[71] Lefei Zhang, Liangpei Zhang, Dacheng Tao, Xin Huang, and Bo Du.Compression of hyperspectral remote sensing images by tensor approach.Neurocomputing, 147:358-363, 2015.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.