×

Fast and separable estimation in high-dimensional tensor Gaussian graphical models. (English) Zbl 07546477

Summary: In the tensor data analysis, the Kronecker covariance structure plays a vital role in unsupervised learning and regression. Under the Kronecker covariance model assumption, the covariance of an \(M\)-way tensor is parameterized as the Kronecker product of \(M\) individual covariance matrices. With normally distributed tensors, the key to high-dimensional tensor graphical models becomes the sparse estimation of the \(M\) inverse covariance matrices. Unable to maximize the tensor normal likelihood analytically, existing approaches often require cyclic updates of the \(M\) sparse matrices. For the high-dimensional tensor graphical models, each update step solves a regularized inverse covariance estimation problem that is computationally nontrivial. This computational challenge motivates our study of whether a noncyclic approach can be as good as the cyclic algorithms in theory and practice. To handle the potentially very high-dimensional and high-order tensors, we propose a separable and parallel estimation scheme. We show that the new estimator achieves the same minimax optimal convergence rate as the cyclic estimation approaches. Numerically, the new estimator is much faster and often more accurate than the cyclic approach. Moreover, another advantage of the separable estimation scheme is its flexibility in modeling, where we can easily incorporate user-specified or specially structured covariances on any modes of the tensor. We demonstrate the efficiency of the proposed method through both simulations and a neuroimaging application. Supplementary materials for this article are available online.

MSC:

62-XX Statistics
Full Text: DOI

References:

[1] Banerjee, O.; Ghaoui, L. E.; d’Aspremont, A., “Model Selection Through Sparse Maximum Likelihood Estimation for Multivariate Gaussian or Binary Data,”, Journal of Machine Learning Research, 9, 485-516 (2008) · Zbl 1225.68149
[2] Bi, X.; Tang, X.; Yuan, Y.; Zhang, Y.; Qu, A., “Tensors in Statistics, Annual Review of Statistics and Its Application, 8, 345-336 (2021) · doi:10.1146/annurev-statistics-042720-020816
[3] Cai, T. T.; Liu, W.; Zhou, H. H., “Estimating Sparse Precision Matrix: Optimal Rates of Convergence and Adaptive Estimation, The Annals of Statistics, 44, 455-488 (2016) · Zbl 1341.62115 · doi:10.1214/13-AOS1171
[4] Chi, E. C.; Kolda, T. G., “On Tensors, Sparsity, and Nonnegative Factorizations, SIAM Journal on Matrix Analysis and Applications, 33, 1272-1299 (2012) · Zbl 1262.15029 · doi:10.1137/110859063
[5] Danaher, P.; Wang, P.; Witten, D. M., “The Joint Graphical Lasso for Inverse Covariance Estimation Across Multiple Classes, Journal of the Royal Statistical Society, Series B, 76, 373 (2014) · Zbl 07555455 · doi:10.1111/rssb.12033
[6] Drton, M., Kuriki, S., and Hoff, P. (2020), “Existence and Uniqueness of the Kronecker Covariance MLE,” arXiv no. 2003.06024. · Zbl 1480.62096
[7] Friedman, J.; Hastie, T.; Tibshirani, R., “Sparse Inverse Covariance Estimation With the Graphical Lasso, Biostatistics, 9, 432-441 (2008) · Zbl 1143.62076 · doi:10.1093/biostatistics/kxm045
[8] Gupta, A. K.; Nagar, D. K., Matrix Variate Distributions (2018), Boca Ration: Chapman and Hall/CRC, Boca Ration · Zbl 0935.62064
[9] He, S.; Yin, J.; Li, H.; Wang, X., “Graphical Model Selection and Estimation for High Dimensional Tensor Data, Journal of Multivariate Analysis, 128, 165-185 (2014) · Zbl 1352.62081 · doi:10.1016/j.jmva.2014.03.007
[10] Hoff, P. D., “Multilinear Tensor Regression for Longitudinal Relational Data, The Annals of Applied Statistics, 9, 1169-1193 (2015) · Zbl 1454.62481 · doi:10.1214/15-AOAS839
[11] Kolda, T. G.; Bader, B. W., “Tensor Decompositions and Applications, SIAM Review, 51, 455-500 (2009) · Zbl 1173.65029 · doi:10.1137/07070111X
[12] Lauritzen, S. L., Graphical Models, 17 (1996), Oxford: Clarendon Press, Oxford · Zbl 0907.62001
[13] Lei, X.; Liao, K., “Understanding the Influences of Eeg Reference: A Large-Scale Brain Network Perspective, Frontiers in Neuroscience, 11, 205 (2017) · doi:10.3389/fnins.2017.00205
[14] Leng, C.; Tang, C. Y., “Sparse Matrix Graphical Models, Journal of the American Statistical Association, 107, 1187-1200 (2012) · Zbl 1443.62194 · doi:10.1080/01621459.2012.706133
[15] Li, B.; Kim, M. K.; Altman, N., “On Dimension Folding of Matrix-or Array-Valued Statistical Objects, The Annals of Statistics, 38, 1094-1121 (2010) · Zbl 1183.62091 · doi:10.1214/09-AOS737
[16] Lock, E. F., “Tensor-on-Tensor Regression, Journal of Computational and Graphical Statistics, 27, 638-647 (2018) · Zbl 07498939 · doi:10.1080/10618600.2017.1401544
[17] Lyu, X.; Sun, W. W.; Wang, Z.; Liu, H.; Yang, J.; Cheng, G., Tensor Graphical Model: Non-Convex Optimization and Statistical Inference, IEEE Transactions on Pattern Analysis and Machine Intelligence, 42 (2019) · doi:10.1109/TPAMI.2019.2907679
[18] Molstad, A. J.; Rothman, A. J., “Shrinking Characteristics of Precision Matrix Estimators, Biometrika, 105, 563-574 (2018) · Zbl 1499.62179 · doi:10.1093/biomet/asy023
[19] Pan, Y.; Mai, Q.; Zhang, X., “Covariate-Adjusted Tensor Classification in High Dimensions, Journal of the American Statistical Association, 114, 1305-1319 (2019) · Zbl 1428.62291 · doi:10.1080/01621459.2018.1497500
[20] Pfeiffer, R. M.; Kapla, D. B.; Bura, E., “Least Squares and Maximum Likelihood Estimation of Sufficient Reductions in Regressions With Matrix-Valued Predictors, International Journal of Data Science and Analytics, 11, 11-26 (2021) · doi:10.1007/s41060-020-00228-y
[21] Ravikumar, P.; Wainwright, M. J.; Raskutti, G.; Yu, B., “High-Dimensional Covariance Estimation by Minimizing l1-Penalized Log-Determinant Divergence, Electronic Journal of Statistics, 5, 935-980 (2011) · Zbl 1274.62190 · doi:10.1214/11-EJS631
[22] Rothman, A. J.; Bickel, P. J.; Levina, E.; Zhu, J., “Sparse Permutation Invariant Covariance Estimation, Electronic Journal of Statistics, 2, 494-515 (2008) · Zbl 1320.62135 · doi:10.1214/08-EJS176
[23] Tsiligkaridis, T.; Hero III, A. O.; Zhou, S., “On Convergence of Kronecker Graphical Lasso Algorithms, IEEE Transactions on Signal Processing, 61, 1743-1755 (2013) · Zbl 1394.62095 · doi:10.1109/TSP.2013.2240157
[24] Tzambazis, K.; Stough, C., “Alcohol Impairs Speed of Information Processing and Simple and Choice Reaction Time and Differentially Impairs Higher-Order Cognitive Abilities, Alcohol and Alcoholism, 35, 197-201 (2000) · doi:10.1093/alcalc/35.2.197
[25] Witten, D. M.; Friedman, J. H.; Simon, N., “New Insights and Faster Computations for the Graphical Lasso, Journal of Computational and Graphical Statistics, 20, 892-900 (2011) · doi:10.1198/jcgs.2011.11051a
[26] Xu, P.; Zhang, T.; Gu, Q.; Singh, Aarti; Zhu, Jerry, Artificial Intelligence and Statistics, Efficient Algorithm for Sparse Tensor-Variate Gaussian Graphical Models Via Gradient Descent, 923-932 (2017)
[27] Yin, J.; Li, H., “Model Selection and Estimation in the Matrix Normal Graphical Model, Journal of Multivariate Analysis, 107, 119-140 (2012) · Zbl 1236.62058 · doi:10.1016/j.jmva.2012.01.005
[28] Yuan, M.; Lin, Y., “Model Selection and Estimation in the Gaussian Graphical Model, Biometrika, 94, 19-35 (2007) · Zbl 1142.62408 · doi:10.1093/biomet/asm018
[29] Zhang, T.; Zou, H., “Sparse Precision Matrix Estimation Via Lasso Penalized d-Trace Loss, Biometrika, 101, 103-120 (2014) · Zbl 1285.62063 · doi:10.1093/biomet/ast059
[30] Zhang, X. L.; Begleiter, H.; Porjesz, B.; Wang, W.; Litke, A., “Event Related Potentials During Object Recognition Tasks, Brain Research Bulletin, 38, 531-538 (1995) · doi:10.1016/0361-9230(95)02023-5
[31] Zhou, H.; Li, L.; Zhu, H., “Tensor Regression With Applications in Neuroimaging Data Analysis, Journal of the American Statistical Association, 108, 540-552 (2013) · Zbl 06195959 · doi:10.1080/01621459.2013.776499
[32] Zhou, S., “Gemini: Graph Estimation With Matrix Variate Normal Instances, The Annals of Statistics, 42, 532-562 (2014) · Zbl 1301.62054 · doi:10.1214/13-AOS1187
[33] Zhu, Y.; Li, L., “Multiple Matrix Gaussian Graphs Estimation, Journal of the Royal Statistical Society, Series B, 80, 927-850 (2018) · Zbl 1407.62214 · doi:10.1111/rssb.12278
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.