Learning subspaces of different dimensions. (English) Zbl 07547615
Summary: We introduce a Bayesian model for inferring mixtures of subspaces of different dimensions. The model allows flexible and efficient learning of a density supported in an ambient space which in fact can concentrate around some lower-dimensional space. The key challenge in such a mixture model is specification of prior distributions over subspaces of different dimensions. We address this challenge by embedding subspaces or Grassmann manifolds into a sphere of relatively low dimension and specifying priors on the sphere. We provide an efficient sampling algorithm for the posterior distribution of the model parameters. We illustrate that a simple extension of our mixture of subspaces model can be applied to topic modeling. The utility of our approach is demonstrated with applications to real and simulated data.
MSC:
| 62-XX | Statistics |
References:
| [1] | Amari, S., “Differential Geometry of Curved Exponential Families — Curvatures and Information Loss, Annals of Statistics, 10, 357-385 (1982) · Zbl 0507.62026 |
| [2] | Ashikhmin, A.; Calderbank, A. R., “Grassmannian Packings From Operator Reed-Muller Codes, IEEE Transactions on Information Theory, 56, 5689-5714 (2003) · Zbl 1366.94574 · doi:10.1109/TIT.2010.2070192 |
| [3] | Bache, K.; Lichman, M., “UCI Machine Learning Repository.” (2013) |
| [4] | Banerjee, A.; Dhillon, I. S.; Ghosh, J.; Sra, S., “Clustering on the Unit Hypersphere Using von Mises-Fisher Distributions, Journal of Machine Learning Research, 6, 1345-1382 (2005) · Zbl 1190.62116 |
| [5] | Belkin, M.; Niyogi, P., “Laplacian Eigenmaps for Dimensionality Reduction and Data Representation, Neural Computation, 15, 1373-1396 (2003) · Zbl 1085.68119 · doi:10.1162/089976603321780317 |
| [6] | Bendich, P.; Mukherjee, S.; Wang, B., Local Homology Transfer and Stratification Learning, ACM-SIAM Symposium on Discrete Algorithms (2012) · Zbl 1421.68143 · doi:10.1137/1.9781611973099.107 |
| [7] | Blei, D. M.; Ng, A. Y.; Jordan, M. I., Latent Dirichlet Allocation Journal of Machine Learning Research, 3, 993-1022 (2003) · Zbl 1112.68379 |
| [8] | Carvalho, C. M.; Chang, J.; Lucas, J. E.; Nevins, J. R.; Wang, Q.; West, M., “High-Dimensional Sparse Factor Modeling: Applications in Gene Expression Genomics, Journal of the American Statistical Association, 103, 1438-1456 (2008) · Zbl 1286.62091 · doi:10.1198/016214508000000869 |
| [9] | Chandra, K., Canale, A., and Dunson, D. (2020), “Escaping the Curse of Dimensionality in Bayesian Model Based Clustering,” arXiv e-prints 2006.02700. |
| [10] | Chen, A.; De, A.; Vijayaraghavan, A., Learning a Mixture of Two Subspaces Over Finite Fields, 481-504 (2021) |
| [11] | Conway, J. H.; Hardin, R. H.; Sloane, N. J.A., “Packing Lines, Planes, Etc.: Packings in GRassmannian Spaces,”, Experiment. Math, 5, 83-159 (1996) · Zbl 0864.51012 |
| [12] | Cook, R., “Fisher Lecture: Dimension Reduction in Regression, Statistical Science, 22, 1 (2007) · Zbl 1246.62148 · doi:10.1214/088342306000000682 |
| [13] | Detrano, R.; Janosi, A.; Steinbrunn, W.; Pfisterer, M.; Schmid, J.; Sandhu, S.; Guppy, K.; Lee, S.; Froelicher, V., “International Application of a New Probability Algorithm for the Diagnosis of Coronary Artery Disease, American Journal of Cardiology, 604, 304-310 (1989) · doi:10.1016/0002-9149(89)90524-9 |
| [14] | Deerwester, S.; Dumais, S. T.; Furnas, G. W.; Landauer, T. K.; Harshman, R., Indexing by latent semantic analysis, Journal of the American Society for Information Science, 41, 391-407 (1990) · doi:10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9 |
| [15] | Donoho, D.; Grimes, C., “Hessian Eigenmaps: New Locally Linear Embedding Techniques for High-Dimensional Data, Proceedings of the National Academy of Sciences, 100, 5591-5596 (2003) · Zbl 1130.62337 · doi:10.1073/pnas.1031596100 |
| [16] | Elhamifar, E.; Vidal, R., “Sparse Subspace Clustering,”, IEEE Conference on Computer Vision and Pattern Recognition, 2790-2797 (2009) |
| [17] | Efron, B., “The Geometry of Exponential Families, Annals of Statistics, 6, 362-376 (1978) · Zbl 0436.62027 |
| [18] | Fisher, R. A., “Dispersion on a Sphere, 217, 295-305 (1953) · Zbl 0051.37105 |
| [19] | Dan Geiger, D. H.; King, H.; Meek, C., “Stratified Exponential Families: Graphical Models and Model Selection, Annals of Statistics, 29, 505-529 (2001) · Zbl 1012.62012 |
| [20] | Giné, E.; Koltchinskii, V., High-Dimensional Probability, Vol. 51 of IMS Lecture Notes Monogr. Ser, “Empirical Graph Laplacian Approximation of Laplace-Beltrami Operators: Large Sample Results,”, 238-259 (2006), Beachwood, OH: Inst. Math. Statist, Beachwood, OH · Zbl 1124.60030 |
| [21] | Golub, G.; Van Loan, C., Matrix Computations (2013), Baltimore, MD: John Hopkins University Press, Baltimore, MD · Zbl 1268.65037 |
| [22] | Goresky, M.; MacPherson, R., Stratified Morse Theory (1988), Springer-Verlag · Zbl 0639.14012 |
| [23] | Green, P. J., “Reversible Jump Markov Chain Monte Carlo Computation and Bayesian Model Determination, Biometrika, 82, 711-732 (1995) · Zbl 0861.62023 · doi:10.1093/biomet/82.4.711 |
| [24] | Hamm, J.; Lee, D. D., “Grassmann Discriminant Analysis: A Unifying View on Subspace-Based Learning, Advances in NIPS, 17 (2005) |
| [25] | Hansen, T. F.; Houle, D., “Measuring and Comparing Evolvability and Constraint in Multivariate Characters, Journal of Evolutionary Biology, 21, 1201-1219 (2008) · doi:10.1111/j.1420-9101.2008.01573.x |
| [26] | Haro, G.; Randall, G.; Sapiro, G., “Stratification Learning: Detecting Mixed Density and Dimensionality in High Dimensional Point Clouds,”, Advances in Neural Information Processing Systems, 19, 553-560 (2007) |
| [27] | Hoff, P. D., “Simulation of the Matrix Bingham-von Mises-Fisher Distribution, With Applications to Multivariate and Relational Data, The Journal of Computational and Graphical Statistics, 18, 438-456 (2009) · doi:10.1198/jcgs.2009.07177 |
| [28] | Hoffman, T., Probabilistic Latent Semantic Indexing, 50-57 (1999) |
| [29] | Huang, K.; Ma, Y.; Vidal, R., “Minimum Effective Dimension for Mixtures of Subspaces: A Robust GPCA Algorithm and Its Applications, IEEE Conference on Computer Vision and Pattern Recognition, II, 631-638 (1999) |
| [30] | Jiang, W.; Tanner, M. A., “Gibbs Posterior for Variable Selection in High-Dimensional Classification and Data Mining,”, Annals of Statistics, 36, 2025-2550 (2008) |
| [31] | Kendall, D. G., “Shape Manifolds, Procrustean Metrics, and Complex Projective Spaces, Bulletin of the London Mathematical Society, 16, 81-121 (1984) · Zbl 0579.62100 · doi:10.1112/blms/16.2.81 |
| [32] | Laaksone, J.; Oja, E., Subspace Dimension Selection and Averaged Learning Subspace Method in Handwritten Digit Classification, 227-232 (1996) |
| [33] | Lande, R., “Quantitative Genetic-Analysis of Multivariate Evolution, Applied to Brain-Body Size Allometry, Evolution, 33, 402-416 (1979) · doi:10.2307/2407630 |
| [34] | Lerman, G.; Zhang, T., “Probabilistic Recovery of Multiple Subspaces in Point Clouds by Geometric lp Minimization,”, Annals of Statistics, 39, 2686-2715 (2010) · Zbl 1232.62097 |
| [35] | Lipor, J., Hong, D., Tan, Y. S., and Balzano, L. (2021), “Subspace Clustering Using Ensembles of K-Subspaces,” arXiv: 1709.04744. · Zbl 1472.62098 |
| [36] | Liu, G.; Lin, Z.; Yu, Y., Robust Subspace Segmentation by Low-Rank Representation, 663-670 (2010) |
| [37] | Lu, C.; Min, H.; Zhao, Z.; Zhu, L.; Huang, D.; Yan, S., Robust and Efficient Subspace Segmentation Via Least Squares Regression, European Conference on Computer Vision, 7578, 347-360 (2012) |
| [38] | Mangasarian, O. L.; Wolberg, W. H., Cancer Diagnosis Via Linear Programming, Siam News, 23, 1 (1990) · Zbl 0726.90096 |
| [39] | McCallum, A. K., MALLET: A Machine Learning for Language Toolkit (2002) |
| [40] | Mukherjee, S.; Zhou, D-X.; Wu, Q., “Learning Gradients and Feature Selection on Manifolds, Bernoulli, 16, 181-207 (2010) · Zbl 1200.62070 · doi:10.3150/09-BEJ206 |
| [41] | NSF 2010 Awards. Available at: |
| [42] | Page, G.; Bhattacharya, A.; Dunson, D. B., “Classification Via Bayesian Nonparametric Learning of Affine Subspaces, Journal of American Statistical Association, 108, 187-201 (2013) · Zbl 06158335 · doi:10.1080/01621459.2013.763566 |
| [43] | Pimentel-Alarcón, D.; Balzano, L.; Marcia, R.; Nowak, R.; Willett, R., Mixture Regression as Subspace Clustering, 2017 International Conference on Sampling Theory and Applications (SampTA), Tallin, 456-459 (2017) · doi:10.1109/SAMPTA.2017.8024386 |
| [44] | Pritchard, J. K.; Stephens, M.; Donnelly, P., “Inference of Population Structure Using Multilocus Genotype Data, Genetics, 155, 945-959 (2000) · doi:10.1093/genetics/155.2.945 |
| [45] | Rao, C. R., “Information and Accuracy Obtainable in the Estimation of Statistical Parameters, Bulletin Calcutta Mathematical Society, 37, 81-91 (1945) · Zbl 0063.06420 |
| [46] | Reisinger, J.; Waters, A.; Silverthorn, B.; Mooney, R. J., Spherical Topic Models (2010) |
| [47] | Roweis, S.; Saul, L., “Nonlinear Dimensionality Reduction by Locally Linear Embedding, Science, 290, 2323-2326 (2000) · doi:10.1126/science.290.5500.2323 |
| [48] | Schwartz, L., “On Bayes Procedures, Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete, 4, 10-26 (1965) · Zbl 0158.17606 · doi:10.1007/BF00535479 |
| [49] | Siebert, J. P., Turing Institute Research Memorandum TIRM-87-018, Vehicle Recognition Using Rule Based Methods (1987), Glasgow, Scotland: Turing Institute, Glasgow, Scotland |
| [50] | Soltanolkotabi, M.; Candés, E., “A Geometric Analysis of Subspace Clustering With Outliers,”, Annals of Statistics, 40, 2195-2238 (2012) · Zbl 1318.62217 |
| [51] | Vidal, R.; Ma, Y.; Sastry, S., “Generalized Principal Component Analysis (GPCA), IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 1945-1959 (2005) · doi:10.1109/TPAMI.2005.244 |
| [52] | Vidal, R.; Favaro, P., “Low Rank Subspace Clustering (LRSC), Pattern Recognition Letters, 43, 47-61 (2014) · doi:10.1016/j.patrec.2013.08.006 |
| [53] | Wu, H. C.; Luk, R. W. P.; Wong, K. F.; Kwok, K. L., “Interpreting TF-IDF Term Weights As Making Relevance Decisions, ACM Transactions on Information Systems, 26, 1-37 (2008) · doi:10.1145/1361684.1361686 |
| [54] | Zheng, L.; Tse, D. N. C., “Communication on the Grassmann Manifold: A Geometric Approach to the Noncoherent Multiple-Antena Channel, IEEE Transactions on Information Theory, 48, 359-383 (2002) · Zbl 1071.94503 · doi:10.1109/18.978730 |
| [55] | Zheng, H.; Wang, Y.; Li, T., Foundations of Intelligent Systems. Advances in Intelligent and Soft Computing, 122, Mixture of Subspace Learning With Adaptive Dimensionality: A Self-Organizing Approach (2011), Berlin: Springer, Berlin |
| [56] | Zhong, S.; Ghosh, J., “Generative Model-Based Document Clustering: A Comparative Study, Knowledge and Information Systems, 8, 374-384 (2005) · doi:10.1007/s10115-004-0194-1 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
