Locally discriminative topic modeling. (English) Zbl 1225.68221
Summary: Topic modeling is a powerful tool for discovering the underlying or hidden structure in text corpora. Typical algorithms for topic modeling include probabilistic latent semantic analysis (PLSA) and latent Dirichlet allocation (LDA). Despite their different inspirations, both approaches are instances of generative model, whereas the discriminative structure of the documents is ignored. In this paper, we propose locally discriminative topic model (LDTM), a novel topic modeling approach which considers both generative and discriminative structures of the data space. Different from PLSA and LDA in which the topic distribution of a document is dependent on all the other documents, LDTM takes a local perspective that the topic distribution of each document is strongly dependent on its neighbors. By modeling the local relationships of documents within each neighborhood via a local linear model, we learn topic distributions that vary smoothly along the geodesics of the data manifold, and can better capture the discriminative structure in the data. The experimental results on text clustering and web page categorization demonstrate the effectiveness of our proposed approach.
MSC:
| 68T05 | Learning and adaptive systems in artificial intelligence |
Software:
ElemStatLearnReferences:
| [1] | M. Belkin, I. Matveeva, P. Niyogi, Regularization and semi-supervised learning on large graphs, in: Proceedings of the 17th Conference on Learning Theory (COLT), 2004, pp. 624-638.; M. Belkin, I. Matveeva, P. Niyogi, Regularization and semi-supervised learning on large graphs, in: Proceedings of the 17th Conference on Learning Theory (COLT), 2004, pp. 624-638. · Zbl 1078.68685 |
| [2] | Belkin, M.; Niyogi, P., Towards a theoretical foundation for Laplacian-based manifold methods, Journal of Computer and System Sciences, 74, 8, 1289-1308 (2008) · Zbl 1157.68056 |
| [3] | Belkin, M.; Niyogi, P., Laplacian eigenmaps for dimensionality reduction and data representation, Neural Computation, 15, 6, 1373-1396 (2003) · Zbl 1085.68119 |
| [4] | Belkin, M.; Niyogi, P.; Sindhwani, V., Manifold regularization: a geometric framework for learning from labeled and unlabeled examples, Journal of Machine Learning Research, 7, November, 2399-2434 (2006) · Zbl 1222.68144 |
| [5] | Bottou, L.; Vapnik, V., Local learning algorithms, Neural Computation, 4, 888-900 (1992) |
| [6] | O. Chapelle, M. Chi, A. Zien, A continuation method for semi-supervised SVMs, in: Proceedings of the 23rd International Conference on Machine Learning, 2006, pp. 185-192.; O. Chapelle, M. Chi, A. Zien, A continuation method for semi-supervised SVMs, in: Proceedings of the 23rd International Conference on Machine Learning, 2006, pp. 185-192. |
| [7] | Chapelle, O.; Schölkopf, B.; Zien, A., Semi-Supervised Learning (2006), MIT Press: MIT Press Cambridge, MA, USA, p. 508 |
| [8] | Fan, R.-E.; Chen, P.-H.; Lin, C.-J., Working set selection using second order information for training SVM, Journal of Machine Learning Research, 6, 1889-1918 (2005) · Zbl 1222.68198 |
| [9] | Gloub, G. H.; Vanloan, C. F., Matrix Computations (1983), Johns Hopking UP: Johns Hopking UP Baltimore · Zbl 0559.65011 |
| [10] | M. Hein, J.Y. Audibert, von U. Luxburg, From graphs to manifolds-weak and strong pointwise consistency of graph Laplacians, in: Proceedings of the 18th Annual Conference on Learning Theory (COLT), 2005, pp. 470-485.; M. Hein, J.Y. Audibert, von U. Luxburg, From graphs to manifolds-weak and strong pointwise consistency of graph Laplacians, in: Proceedings of the 18th Annual Conference on Learning Theory (COLT), 2005, pp. 470-485. · Zbl 1095.68097 |
| [11] | T. Joachims, Transductive inference for text classification using support vector machines, in: Proceedings of the International Conference on Machine Learning (ICML), 1999, pp. 200-209.; T. Joachims, Transductive inference for text classification using support vector machines, in: Proceedings of the International Conference on Machine Learning (ICML), 1999, pp. 200-209. |
| [12] | Jolliffe, I. T., Principal Component Analysis (2002), Springer: Springer NY · Zbl 1011.62064 |
| [13] | Kapoor, A.; Qi, Y.; Ahn, H.; Picard, R. W., Hyperparameter and kernel learning for graph based semi-supervised classification, In Advances in Neural Information Processing Systems (NIPS), 18, 627-634 (2006) |
| [14] | Lal, T. N.; Schröder, M.; Hinterberger, T.; Weston, J.; Bogdan, M.; Birbaumer, N.; Schölkopf, B., Support vector channel selection in BCI, IEEE Transactions on Biomedical Engineering, 51, 6, 1003-1010 (2004) |
| [15] | Levin, A.; Lischinski, D.; Weiss, Y., A closed form solution to natural image matting, IEEE Transactions Pattern Analysis and Machine Intelligence, 30, 2, 228-242 (2007) |
| [16] | W. Liu, D. Tao, J. Liu, Transductive component analysis, in: Proceedings of the Eighth IEEE International Conference on Data Mining, 2008, pp. 433-442.; W. Liu, D. Tao, J. Liu, Transductive component analysis, in: Proceedings of the Eighth IEEE International Conference on Data Mining, 2008, pp. 433-442. |
| [17] | Roweis, S.; Saul, S., Nonlinear dimensionality reduction by locally linear embedding, Science, 290, 5500, 2323-2326 (2000) |
| [18] | Schölkopf, B.; Smola, A., Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond (2002), The MIT Press: The MIT Press Cambridge, MA |
| [19] | Smola, A. J.; Bartlett, P. L.; Scholkopf, B.; Schuurmans, D., Advances in Large Margin Classifiers (2000), The MIT Press: The MIT Press Cambridge, MA · Zbl 0988.68145 |
| [20] | Rao, C. R.; Mitra, S. K., Generalized Inverse of Matrices (1971), Wiley: Wiley New York · Zbl 0236.15004 |
| [21] | Vapnik, V. N., The Nature of Statistical Learning Theory (1995), Springer-Verlag: Springer-Verlag Berlin · Zbl 0833.62008 |
| [22] | F. Wang, C. Zhang, T. Li, Regularized clustering for documents, in: Proceedings of The 30th International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR), Amsterdam, The Netherlands. 2007, pp. 95-102 .; F. Wang, C. Zhang, T. Li, Regularized clustering for documents, in: Proceedings of The 30th International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR), Amsterdam, The Netherlands. 2007, pp. 95-102 . |
| [23] | F. Wang, C. Zhang, T. Li, Clustering with local and global regularization, in: Proceedings of the 22nd AAAI Conference on Artificial Intelligence (AAAI), Vancouver, Canada, 2007, pp. 657-662.; F. Wang, C. Zhang, T. Li, Clustering with local and global regularization, in: Proceedings of the 22nd AAAI Conference on Artificial Intelligence (AAAI), Vancouver, Canada, 2007, pp. 657-662. |
| [24] | F. Wang, S. Wang, C. Zhang, O. Winther, Semi-supervised mean fields, in: Proceedings of the 11th International Conference on Artificial Intelligence and Statistics (AISTATS), 2007, pp. 596-603.; F. Wang, S. Wang, C. Zhang, O. Winther, Semi-supervised mean fields, in: Proceedings of the 11th International Conference on Artificial Intelligence and Statistics (AISTATS), 2007, pp. 596-603. |
| [25] | F. Wang, T. Li, C. Zhang, Semi-supervised clustering via matrix factorization. in: Proceedings of the 8th SIAM Conference on Data Mining (SDM), Hyatt Regency Hotel, Atlanta, Georgia, 2008, pp. 1-12.; F. Wang, T. Li, C. Zhang, Semi-supervised clustering via matrix factorization. in: Proceedings of the 8th SIAM Conference on Data Mining (SDM), Hyatt Regency Hotel, Atlanta, Georgia, 2008, pp. 1-12. |
| [26] | M. Wu, B. Schölkopf, A local learning approach for clustering, Advances in Neural Information Processing Systems, 2006, pp. 1-8.; M. Wu, B. Schölkopf, A local learning approach for clustering, Advances in Neural Information Processing Systems, 2006, pp. 1-8. |
| [27] | M. Wu, B. Schölkopf, Transductive classification via local learning regularization. in: Proceedings of the 11th International Conference on Artificial Intelligence and Statistics (AISTATS), 2007, pp. 624-631.; M. Wu, B. Schölkopf, Transductive classification via local learning regularization. in: Proceedings of the 11th International Conference on Artificial Intelligence and Statistics (AISTATS), 2007, pp. 624-631. |
| [28] | D. Zhang, F. Wang, C. Zhang, T. Li, Multi-view local learning, in: Proceedings of The 23rd AAAI Conference on Artificial Intelligence (AAAI), Chicago, Illinois, USA, July 13-17, 2008, pp. 752-757.; D. Zhang, F. Wang, C. Zhang, T. Li, Multi-view local learning, in: Proceedings of The 23rd AAAI Conference on Artificial Intelligence (AAAI), Chicago, Illinois, USA, July 13-17, 2008, pp. 752-757. |
| [29] | Zhang, T.; Tao, D.; Li, X.; Yang, J., Patch alignment for dimensionality reduction, IEEE Transactions on Knowledge and Data Engineering, 21, 9, 1299-1313 (2009) |
| [30] | Zhou, D.; Bousquet, O.; Lal, T. N.; Weston, J.; Schölkopf, B., Learning with local and global consistency, Advances in Neural Information Processing Systems, 16, 321-328 (2004) |
| [31] | X. Zhu, Z. Ghahramani, Z. Lafferty, Semi-supervised learning using Gaussian fields and harmonic functions, in: Proceedings of the 20th International Conference on Machine Learning (ICML), 2003, pp. 912-919.; X. Zhu, Z. Ghahramani, Z. Lafferty, Semi-supervised learning using Gaussian fields and harmonic functions, in: Proceedings of the 20th International Conference on Machine Learning (ICML), 2003, pp. 912-919. |
| [32] | X. Zhu, A. Goldberg, Kernel regression with order preferences, in: Proceedings of the Twenty-Second AAAI Conference on Artificial Intelligence (AAAI), 2007.; X. Zhu, A. Goldberg, Kernel regression with order preferences, in: Proceedings of the Twenty-Second AAAI Conference on Artificial Intelligence (AAAI), 2007. |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
