Cauchy noise loss for stochastic optimization of random matrix models via free deterministic equivalents. (English) Zbl 1435.62192
The main contribution of this paper is a parameter optimization method for random matrix models. A dimensionality recovery method for the signal-plus-noise model is proposed. Numerical experiments are included to demonstrate the efficiency of the proposed theory.
Reviewer: Denis Sidorov (Irkutsk)
MSC:
| 62H12 | Estimation in multivariate analysis |
| 15B52 | Random matrices (algebraic aspects) |
| 49J55 | Existence of optimal solutions to problems involving randomness |
| 60G60 | Random fields |
Keywords:
random matrix theory; free probability theory; stochastic optimization; rank estimation; dimensionality recoverySoftware:
AdamReferences:
| [1] | Amari, S., Natural gradient works efficiently in learning, Neural Comput., 10, 2, 251-276 (1998) |
| [2] | Belinschi, S. T.; Mai, T.; Speicher, R., Analytic subordination theory of operator-valued free additive convolution and the solution of a general random matrix problem, J. Reine Angew. Math. (2013) |
| [3] | Billingsley, P., Probability and Measure (2008), John Wiley & Sons |
| [4] | Bottou, L., Online Learning and Stochastic Approximations, 9-42 (1998), Cambridge Univ. Press · Zbl 0968.68127 |
| [5] | Christopher, M. B., Pattern Recognition and Machine Learning (2016), Springer-Verlag: Springer-Verlag New York |
| [6] | Collins, B.; McDonald, D.; Saad, N., Compound Wishart matrices and noisy covariance matrices: risk underestimation (2013), preprint |
| [7] | Collins, B.; Mingo, J. A.; Sniady, P.; Speicher, R., Second order freeness and fluctuations of random matrices. III. Higher order freeness and free cumulants, Doc. Math., 12, 1-70 (2007) · Zbl 1123.46047 |
| [8] | Couillet, R.; Debbah, M.; Silverstein, J. W., A deterministic equivalent for the analysis of correlated MIMO multiple access channels, IEEE Trans. Inform. Theory, 57, 6, 3493-3514 (2011) · Zbl 1365.94123 |
| [9] | Dudley, R. M., Real Analysis and Probability, vol. 74 (2002), Cambridge Univ. Press · Zbl 1023.60001 |
| [10] | Haagerup, U.; Thorbjørnsen, S., A new application of random matrices: \(Ext(C_{\text{red}}^\ast(F_2))\) is not a group, Ann. of Math., 162, 711-775 (2005) · Zbl 1103.46032 |
| [11] | Hachem, W.; Loubaton, P.; Mestre, X.; Najim, J.; Vallet, P., Large information plus noise random matrix models and consistent subspace estimation in large sensor networks, Random Matrices Theory Appl., 1, 02, Article 1150006 pp. (2012) · Zbl 1248.15029 |
| [12] | Hachem, W.; Loubaton, P.; Najim, J., Deterministic equivalents for certain functionals of large random matrices, Ann. Appl. Probab., 17, 3, 875-930 (2007) · Zbl 1181.15043 |
| [13] | Hasegawa, A.; Sakuma, N.; Yoshida, H., Random matrices by MA models and compound free Poisson laws, Probab. Math. Statist., 33, 2, 243-254 (2013) · Zbl 1291.46062 |
| [14] | Hasegawa, A.; Sakuma, N.; Yoshida, H., Fluctuations of Marchenko-Pastur limit of random matrices with dependent entries, Statist. Probab. Lett., 127, 85-96 (2017) · Zbl 1377.60017 |
| [15] | Hayase, T., Free deterministic equivalent z-scores of compound Wishart models: a goodness of fit test of 2DARMA models (2017), preprint |
| [16] | Helton, J. W.; Far, R. R.; Speicher, R., Operator-valued semicircular elements: solving a quadratic matrix equation with positivity constraints, Int. Math. Res. Not., 2007 (2007) · Zbl 1139.15006 |
| [17] | Hiai, F.; Petz, D., The Semicircle Law, Free Random Variables and Entropy, vol. 77 (2006), American Math. Soc. |
| [18] | Kingma, D. P.; Adam, J. B., A method for stochastic optimization, (ICLR (2015)) |
| [19] | Martens, J., New insights and perspectives on the natural gradient method (2014), preprint |
| [20] | Mingo, J. A.; Speicher, R., Free Probability and Random Matrices, vol. 35 (2017), Springer · Zbl 1387.60005 |
| [21] | Nakajima, S.; Sugiyama, M., Theoretical analysis of Bayesian matrix factorization, J. Mach. Learn. Res., 12, 2583-2648 (2011) · Zbl 1280.62034 |
| [22] | Nakajima, S.; Sugiyama, M.; Babacan, S. D.; Tomioka, R., Global analytic solution of fully-observed variational Bayesian matrix factorization, J. Mach. Learn. Res., 14, 1-37 (2013) · Zbl 1436.62228 |
| [23] | Nakajima, S.; Tomioka, R.; Sugiyama, M.; Babacan, S. D., Condition for perfect dimensionality recovery by variational Bayesian PCA, J. Mach. Learn. Res., 16, 3757-3811 (2015) · Zbl 1351.68225 |
| [24] | Nemirovski, A.; Juditsky, A.; Lan, G.; Shapiro, A., Robust stochastic approximation approach to stochastic programming, SIAM J. Optim., 19, 4, 1574-1609 (2009) · Zbl 1189.90109 |
| [25] | Neu, P.; Speicher, R., Rigorous mean-field model for coherent-potential approximation: Anderson model with free random variables, J. Stat. Phys., 80, 5, 1279-1308 (1995) · Zbl 1081.82575 |
| [26] | Nica, A.; Shlyakhtenko, D.; Speicher, R., Operator-valued distributions. I. Characterizations of freeness, Int. Math. Res. Not., 2002, 29, 1509-1538 (2002) · Zbl 1007.46052 |
| [27] | Redelmeier, C. E.I., Real second-order freeness and the asymptotic real second-order freeness of several real matrix models, Int. Math. Res. Not. IMRN, 2014, 12, 3353-3395 (2014) · Zbl 1310.46057 |
| [28] | Robbins, H.; Monro, S., A stochastic approximation method, Ann. Math. Stat., 22, 3, 400-407 (1951) · Zbl 0054.05901 |
| [29] | Ryan, O.; Debbah, M., Free deconvolution for signal processing applications, (IEEE Trans. Inform. Theory (2007), IEEE), 1846-1850 |
| [30] | Speicher, R., Combinatorial Theory of the Free Product with Amalgamation and Operator-Valued Free Probability Theory, vol. 627 (1998), American Math. Soc. · Zbl 0935.46056 |
| [31] | Speicher, R.; Vargas, C., Free deterministic equivalents, rectangular random matrix models, and operator-valued free probability theory, Random Matrices Theory Appl., 1, 02, Article 1150008 pp. (2012) · Zbl 1263.46053 |
| [32] | Tibshirani, R., Regression shrinkage and selection via the lasso, J. Roy. Statist. Soc. Ser. B, 267-288 (1996) · Zbl 0850.62538 |
| [33] | Tipping, M. E.; Bishop, C. M., Probabilistic principal component analysis, J. Roy. Statist. Soc. Ser. B, 61, 3, 611-622 (1999) · Zbl 0924.62068 |
| [34] | Vallet, P.; Loubaton, P.; Mestre, X., Improved subspace estimation for multivariate observations of high dimension: the deterministic signals case, IEEE Trans. Inform. Theory, 58, 2, 1043-1068 (2012) · Zbl 1365.62123 |
| [35] | Vargas, C., Free Probability Theory: Deterministic Equivalents and Combinatorics (2015), Doctoral thesis |
| [36] | Voiculescu, D. V.; Dykema, K. J.; Nica, A., Free Random Variables, vol. 1 (1992), American Math. Soc. · Zbl 0795.46049 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
