Fast moment estimation for generalized latent Dirichlet models. (English) Zbl 1409.62119
Summary: We develop a generalized method of moments (GMM) approach for fast parameter estimation in a new class of Dirichlet latent variable models with mixed data types. Parameter estimation via GMM has computational and statistical advantages over alternative methods, such as expectation maximization, variational inference, and Markov chain Monte Carlo. A key computational advantage of our method, Moment Estimation for latent Dirichlet models (MELD), is that parameter estimation does not require instantiation of the latent variables. Moreover, performance is agnostic to distributional assumptions of the observations. We derive population moment conditions after marginalizing out the sample-specific Dirichlet latent variables. The moment conditions only depend on component mean parameters. We illustrate the utility of our approach on simulated data, comparing results from MELD to alternative methods, and we show the promise of our approach through the application to several datasets.
MSC:
| 62H12 | Estimation in multivariate analysis |
| 62F12 | Asymptotic properties of parametric estimators |
| 62F35 | Robustness and adaptive procedures (parametric inference) |
Keywords:
generalized method of moments; latent Dirichlet allocation; latent variables; mixed membership model; mixed scale data; tensor factorizationReferences:
| [1] | Anandkumar, A.; Ge, R.; Hsu, D.; Kakade, S. M., A Tensor Approach to Learning Mixed Membership Community Models, The Journal of Machine Learning Research, 15, 2239-2312 (2014) · Zbl 1318.68136 |
| [2] | Anandkumar, A.; Ge, R.; Hsu, D.; Kakade, S. M.; Telgarsky, M., Tensor Decompositions for Learning Latent Variable Models, The Journal of Machine Learning Research, 15, 2773-2832 (2014) · Zbl 1319.62109 |
| [3] | Anandkumar, A.; Hsu, D.; Kakade, S. M., A Method of Moments for Mixture Models and Hidden Markov Models, JMLR W&CP 23: COLT (2012) |
| [4] | Anandkumar, A.; Liu, Y.; Hsu, D. J.; Foster, D. P.; Kakade, S. M., A Spectral Algorithm for Latent Dirichlet Allocation, Advances in Neural Information Processing Systems 25, 917-925 (2012) |
| [5] | Anderson, J. C.; Gerbing, D. W., Structural Equation Modeling in Practice: A Review and Recommended Two-Step Approach, Psychological Bulletin, 103, 411 (1988) |
| [6] | Arora, S.; Ge, R.; Moitra, A., Learning Topic Models—Going Beyond SVD, Fifty-Third IEEE Annual Symposium on Foundations of Computer Science, 1-10 (2012) |
| [7] | Bentler, P. M., Some Contributions to Efficient Statistics in Structural Models: Specification and Estimation of Moment Structures, Psychometrika, 48, 493-517 (1983) · Zbl 0533.62091 |
| [8] | Bhattacharya, A.; Dunson, D. B., Simplex Factor Models for Multivariate Unordered Categorical Data, Journal of the American Statistical Association, 107, 362-377 (2012) · Zbl 1263.62097 |
| [9] | Blei, D. M.; Ng, A. Y.; Jordan, M. I., Latent Dirichlet Allocation, The Journal of Machine Learning Research, 3, 993-1022 (2003) · Zbl 1112.68379 |
| [10] | Bollen, K. A.; Kolenikov, S.; Bauldry, S., Model-Implied Instrumental Variable-Generalized Method of Moments (MIIV-GMM) Estimators for Latent Variable Models, Psychometrika, 79, 20-50 (2014) · Zbl 1284.62690 |
| [11] | Colombo, N.; Vlassis, N., FastMotif: Spectral Sequence Motif Discovery, Bioinformatics, 31, 2623-2631 (2015) |
| [12] | Dunson, D. B., Dynamic Latent Trait Models for Multidimensional Longitudinal Data, Journal of the American Statistical Association, 98, 555-563 (2003) · Zbl 1040.62100 |
| [13] | Dunson, D. B.; Xing, C., Nonparametric Bayes Modeling of Multivariate Categorical Data, Journal of the American Statistical Association, 104, 1042-1051 (2009) · Zbl 1388.62151 |
| [14] | Gallant, A. R.; Giacomini, R.; Ragusa, G., Generalized Method of Moments With Latent Variables (2013), Centre for Economic Policy Research |
| [15] | Hall, A. R., Generalized Method of Moments (2005), New York: Oxford University Press, New York · Zbl 1076.62118 |
| [16] | Hansen, L. P., Large Sample Properties of Generalized Method of Moments Estimators, Econometrica: Journal of the Econometric Society, 50, 1029 (1982) · Zbl 0502.62098 |
| [17] | Harley, C. B.; Reynolds, R. P., Analysis of E. coli Promoter Sequences, Nucleic Acids Research, 15, 2343-2361 (1987) |
| [18] | Hoff, P. D., Extending the Rank Likelihood for Semiparametric Copula Estimation, The Annals of Applied Statistics, 1, 265-283 (2007) · Zbl 1129.62050 |
| [19] | Hornik, K.; Grün, B., topicmodels: An R Package for Fitting Topic Models, Journal of Statistical Software, 40, 1-30 (2011) |
| [20] | Hsu, D.; Kakade, S. M., Learning Mixtures of Spherical Gaussians: Moment Methods and Spectral Decompositions, Proceedings of the 4th conference on Innovations in Theoretical Computer Science, 11-20 (2013) · Zbl 1362.68246 |
| [21] | Jöreskog, K. G.; Sörbom, D., New Developments in LISREL, Paper presented at the National Symposium on Methodological Issues in Causal Modeling, University of Alabama, Tuscaloosa (1987) |
| [22] | Kiers, H. A., Towards a Standardized Notation and Terminology in Multiway Analysis, Journal of Chemometrics, 14, 105-122 (2000) |
| [23] | Lichman, M., UCI Machine Learning Repository (2013) |
| [24] | Moustaki, I.; Knott, M., Generalized Latent Trait Models, Psychometrika, 65, 391-411 (2000) · Zbl 1291.62236 |
| [25] | Murray, J. S.; Dunson, D. B.; Carin, L.; Lucas, J. E., Bayesian Gaussian Copula Factor Models for Mixed Data, Journal of the American Statistical Association, 108, 656-665 (2013) · Zbl 06195968 |
| [26] | Muthén, B., A General Structural Equation Model With Dichotomous, Ordered Categorical, and Continuous Latent Variable Indicators, Psychometrika, 49, 115-132 (1984) |
| [27] | Pritchard, J. K.; Stephens, M.; Donnelly, P., Inference of Population Structure Using Multilocus Genotype Data, Genetics, 155, 945-959 (2000) |
| [28] | Pritchard, J. K.; Stephens, M.; Rosenberg, N. A.; Donnelly, P., Association Mapping in Structured Populations, The American Journal of Human Genetics, 67, 170-181 (2000) |
| [29] | Quinn, K. M., Bayesian Factor Analysis for Mixed Ordinal and Continuous Responses, Political Analysis, 12, 338-353 (2004) |
| [30] | Sammel, M. D.; Ryan, L. M.; Legler, J. M., Latent Variable Models for Mixed Discrete and Continuous Outcomes, Journal of the Royal Statistical Society, 59, 667-678 (1997) · Zbl 0889.62043 |
| [31] | Stranger, B. E.; Montgomery, S. B.; Dimas, A. S.; Parts, L.; Stegle, O.; Ingle, C. E.; Sekowska, M.; Smith, G. D.; Evans, D.; Gutierrez-Arcelus, M.; Price, A.; Raj, T.; Nisbett, J.; Nica, A. C.; Beazley, C.; Durbin, R.; Deloukas, P.; Dermitzakis, E. T., Patterns of Cis-Regulatory Variation in Diverse Human Populations, PLoS Genetics, 8, e1002639 (2012) |
| [32] | Tung, H. F.; Smola, A. J., Spectral Methods for Indian Buffet Process Inference, Advances in Neural Information Processing Systems 27, 1484-1492 (2014) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
