Marginal semi-supervised sub-manifold projections with informative constraints for dimensionality reduction and recognition. (English) Zbl 1258.68132
Summary: In this work, sub-manifold projections based semi-supervised dimensionality reduction (DR) problem learning from partial constrained data is discussed. Two semi-supervised DR algorithms termed marginal semi-supervised sub-manifold projections (MS\(^3\)MP) and orthogonal MS\(^3\)MP(OMS\(^3\)MP) are proposed. MS\(^3\)MP in the singular case is also discussed. We also present the weighted least squares view of MS\(^3\)MP. Based on specifying the types of neighborhoods with pairwise constraints (PC) and the defined manifold scatters, our methods can preserve the local properties of all points and discriminant structures embedded in the localized PC. The sub-manifolds of different classes can also be separated. In PC guided methods, exploring and selecting the informative constraints is challenging and random constraint subsets significantly affect the performance of algorithms. This paper also introduces an effective technique to select the informative constraints for DR with consistent constraints. The analytic form of the projection axes can be obtained by eigen-decomposition. The connections between this work and other related work are also elaborated. The validity of the proposed constraint selection approach and DR algorithms are evaluated by benchmark problems. Extensive simulations show that our algorithms can deliver promising results over some widely used state-of-the-art semi-supervised DR techniques.
MSC:
| 68T05 | Learning and adaptive systems in artificial intelligence |
| 68T10 | Pattern recognition, speech recognition |
Keywords:
semi-supervised learning; marginal projections; dimensionality reduction; informative constraints; image recognitionSoftware:
darchReferences:
| [1] | Baghshah, M. S., & Shouraki, S. B. (2009). Semi-supervised metric learning using pairwise constraints. In Proceedings of IJCAI 2009; Baghshah, M. S., & Shouraki, S. B. (2009). Semi-supervised metric learning using pairwise constraints. In Proceedings of IJCAI 2009 |
| [2] | Baghshah, M. S.; Shouraki, S. B., Non-linear metric learning using pairwise similarity and dissimilarity constraints and the geometrical structure of data, Pattern Recognition, 43, 2982-2992 (2010) · Zbl 1207.68261 |
| [3] | Belkin, M.; Niyogi, P., Laplacian eigenmaps for dimensionality reduction and data representation, Neural Computation, 15, 6, 1373-1396 (2003) · Zbl 1085.68119 |
| [4] | Cai, D., He, X. F., & Han, J. (2007). Semi-supervised discriminant analysis. In: Proceedings of ICCV 2007; Cai, D., He, X. F., & Han, J. (2007). Semi-supervised discriminant analysis. In: Proceedings of ICCV 2007 |
| [5] | (Chapelle, O.; Schölkopf, B.; Zien, A., Semi-supervised learning (2006), MIT Press: MIT Press Cambridge) |
| [6] | Chen, H.; Li, L. Q.; Peng, J. T., Semi-supervised learning based on high density region estimation, Neural Networks, 23, 812-818 (2010) · Zbl 1396.68089 |
| [7] | Chen, L.; Liao, H.; Ko, M.; Lin, J.; Yu, G., A new LDA-based face recognition system which can solve the small sample size problem, Pattern Recognition, 33, 10, 1713-1726 (2000) |
| [8] | Chen, J., Ye, J., & Li, Q. (2007). Integrating global and local structures: a least squares framework for dimensionality reduction. In Proceedings of CVPR 2007; Chen, J., Ye, J., & Li, Q. (2007). Integrating global and local structures: a least squares framework for dimensionality reduction. In Proceedings of CVPR 2007 |
| [9] | He, X., Cai, D., Yan, S., & Zhang, H. (2005). Neighborhood preserving embedding. In: Proceedings of ICCV 2005; He, X., Cai, D., Yan, S., & Zhang, H. (2005). Neighborhood preserving embedding. In: Proceedings of ICCV 2005 |
| [10] | He, X.; Yan, S.; Hu, Y.; Niyogi, P.; Zhang, H., Face recognition using Laplacian faces, IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 3, 228-340 (2005) |
| [11] | Hinton, G. E.; Salakhutdinov, R. R., Reducing the dimensionality of data with neural networks, Science, 313, 5786, 504-507 (2006) · Zbl 1226.68083 |
| [12] | Hull, J., A database for handwritten text recognition research, IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 5, 550-554 (1994) |
| [13] | Jia, Y.; Nie, F.; Zhang, C., Trace ratio problem revisited, IEEE Transactions on Neural Networks, 20, 4, 729-735 (2009) |
| [14] | Kokiopoulou1, E.; Chen, J.; Saad, Y., Trace optimization and eigenproblems in dimension reduction methods, Numerical Linear Algebra with Applications, 18, 565-602 (2011) · Zbl 1249.65075 |
| [15] | Leibe, B., & Schiele, B. (2003). Analyzing appearance and contour based methods for object categorization. In Proceedings of CVPR 2003; Leibe, B., & Schiele, B. (2003). Analyzing appearance and contour based methods for object categorization. In Proceedings of CVPR 2003 |
| [16] | Li, H.; Jiang, T.; Zhang, K., Efficient and robust feature extraction by maximum margin criterion, IEEE Transactions on Neural Networks, 17, 1, 157-165 (2006) |
| [17] | Martinez, A. M.; Kak, A. C., PCA versus LDA, IEEE Transactions on Pattern Analysis and Machine Intelligence, 23, 2, 228-233 (2001) |
| [18] | User’s guide (1994-2001), The MathWorks, Inc., MATLAB · Zbl 0992.65013 |
| [19] | Mohammad, N. T. (2007). Dimensionality reduction using neural networks. In Proceedings of the artificial neural networks in engineering, conference; Mohammad, N. T. (2007). Dimensionality reduction using neural networks. In Proceedings of the artificial neural networks in engineering, conference |
| [20] | Qi, Z. Q.; Tian, Y. J.; Shi, Y., Laplacian twin support vector machine for semi-supervised classification, Neural Networks, 35, 46-53 (2012) · Zbl 1258.68121 |
| [21] | Roweis, S.; Saul, L., Nonlinear dimensionality reduction by locally linear embedding, Science, 290, 5500, 2323-2326 (2000) |
| [22] | Schölkopf, B.; Smola, A., Learning with kernels (2002), MIT Press: MIT Press Cambridge, MA, pp. 25-55 |
| [23] | Song, Y. Q.; Nie, F. P.; Zhang, C. S., Semi-supervised sub-manifold discriminant analysis, Pattern Recognition Letters, 29, 1806-1813 (2008) |
| [24] | Song, Y. Q.; Nie, F. P.; Zhang, C. S.; Xiang, S. M., A unified framework for semi-supervised dimensionality reduction, Pattern Recognition, 41, 9, 2789-2799 (2008) · Zbl 1154.68501 |
| [25] | Strutz, T., Data fitting and uncertainty: a practical introduction to weighted least squares and beyond (2010), Vieweg + Teubner Verlag |
| [26] | Sugiyama, M., Dimensionality reduction of multimodal labeled data by local Fisher discriminant analysis, Journal of Machine Learning Research, 8, 1027-1061 (2007) · Zbl 1222.68312 |
| [27] | Sugiyama, M.; Idé, T.; Nakajima, S.; Sese, J., Semi-supervised local Fisher discriminant analysis for dimensionality reduction, Machine Learning, 78, 1-2, 35-61 (2010) · Zbl 1470.68180 |
| [28] | Sun, L., Ceran, B., & Ye, J.P. (2010). A scalable two-stage approach for a class of dimensionality reduction techniques. In Proceedings of ACM KDD 2010; Sun, L., Ceran, B., & Ye, J.P. (2010). A scalable two-stage approach for a class of dimensionality reduction techniques. In Proceedings of ACM KDD 2010 |
| [29] | Sun, L., Ji, S. W., & Ye, J.P. (2008). A least squares formulation for canonical correlation analysis. In Proceedings of ICML 2008; Sun, L., Ji, S. W., & Ye, J.P. (2008). A least squares formulation for canonical correlation analysis. In Proceedings of ICML 2008 |
| [30] | Sun, L.; Ji, S. W.; Ye, J. P., Canonical correlation analysis for multi-label classification: a least squares formulation, extensions and analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, 33, 1, 194-200 (2011) |
| [31] | Sun, D.; Zhang, D. Q., Bagging constraint score for feature selection with pairwise constraints, Pattern Recognition, 43, 2106-2118 (2010) · Zbl 1192.68611 |
| [32] | Venna, J.; Kaski, S., Local multidimensional scaling, Neural Networks, 19, 889-899 (2006) · Zbl 1102.68601 |
| [33] | Weyrauch, B., Huang, J., Heisele, B., & Blanz, V. (2004). Component-based face recognition with 3D morphable models. In Proceedings of the 1st IEEE workshop on face processing in video; Weyrauch, B., Huang, J., Heisele, B., & Blanz, V. (2004). Component-based face recognition with 3D morphable models. In Proceedings of the 1st IEEE workshop on face processing in video · Zbl 1050.68756 |
| [34] | Xiang, S. M.; Nie, F. P.; Zhang, C. S., Learning a Mahalanobis distance metric for data clustering and classification, Pattern Recognition Letters, 41, 12, 3600-3612 (2008) · Zbl 1162.68642 |
| [35] | Xing, E. P.; Ng, A. Y.; Jordan, M. I.; Russell, S., Distance metric learning with application to clustering with side-information, (Proceedings of NIPS 15 (2003), MIT Press: MIT Press Cambridge, MA), 505-512 |
| [36] | Xua, Y.; Zhong, A. N.; Yang, J.; Zhang, D., LPP solution schemes for use with face recognition, Pattern Recognition, 43, 12, 4165-4176 (2010) · Zbl 1207.68329 |
| [37] | Ye, J. P. (2007). Least square linear discriminant analysis. In Proceedings of ICML 2007; Ye, J. P. (2007). Least square linear discriminant analysis. In Proceedings of ICML 2007 |
| [38] | Zhang, D. Q.; Chen, S. C.; Zhou, Z. H., Constraint score: a new filter method for feature selection with pairwise constraints, Pattern Recognition Letters, 41, 5, 1440-1451 (2008) · Zbl 1140.68490 |
| [39] | Zhang, Z.; Chow, T. W.S.; Zhao, M. B., Trace ratio optimization based semi-supervised nonlinear dimensionality reduction for marginal manifold visualization, The IEEE Transactions on Knowledge and Data Engineering (2012) |
| [40] | Zhang, Z.; Zhao, M. B.; Chow, T. W.S., Constrained large margin local projection algorithms and extensions for multimodal dimensionality reduction, Pattern Recognition, 45, 12, 4466-4493 (2012) · Zbl 1248.68426 |
| [41] | Zhang, Z., Zhao, M. B., & Chow, T. W. S. (2012b). Extracting the informative constraints for semi-supervised marginal projections in multimodal dimensionality reduction. In Proceedings of IEEE international joint conference on neural networks; Zhang, Z., Zhao, M. B., & Chow, T. W. S. (2012b). Extracting the informative constraints for semi-supervised marginal projections in multimodal dimensionality reduction. In Proceedings of IEEE international joint conference on neural networks · Zbl 1258.68132 |
| [42] | Zhang, D. Q., Zhou, Z. H., & Chen, S. C. (2007). Semi-supervised dimensionality reduction. In: Proceedings of SDM 2007; Zhang, D. Q., Zhou, Z. H., & Chen, S. C. (2007). Semi-supervised dimensionality reduction. In: Proceedings of SDM 2007 |
| [43] | Zhao, M. B.; Zhang, Z.; Chow, W. S., Trace ratio criterion based generalized discriminative learning for semi-supervised dimensionality reduction, Pattern Recognition, 45, 4, 1482-1499 (2012) · Zbl 1231.68226 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
