Feature extraction framework based on contrastive learning with adaptive positive and negative samples. (English) Zbl 07751945
Summary: Feature extraction is an efficient approach for alleviating the issue of dimensionality in high-dimensional data. As a popular self-supervised learning method, contrastive learning has recently garnered considerable attention. In this study, we propose a unified feature extraction framework based on contrastive learning with adaptive positive and negative samples (CL-FEFA) that is suitable for unsupervised, supervised, and semi-supervised feature extraction. CL-FEFA constructs adaptively positive and negative samples from the result of feature extraction, which makes them more appropriate and accurate. Meanwhile, the discriminative features are extracted based on adaptive positive and negative samples, which will make the intra-class embedded samples more compact and the inter-class embedded samples more dispersed. In the process, using the potential structure information of subspace samples to dynamically construct positive and negative samples can make our framework more robust to noisy data. Furthermore, it is proven that CL-FEFA actually maximizes the mutual information of positive samples, which captures non-linear statistical dependencies between similar samples in potential structure space and thus can act as a measure of true dependence. This also provides theoretical support for its advantages in feature extraction. The final numerical experiments prove that the proposed framework has a strong advantage over traditional feature extraction methods and contrastive learning methods.
MSC:
| 68T05 | Learning and adaptive systems in artificial intelligence |
References:
| [1] | Belhumeur, P. N.; Hespanha, J. P.; Kriegman, D. J., Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection, IEEE Transactions on Pattern Analysis and Machine Intelligence, 19, 7, 711-720 (1997) |
| [2] | Belous, G.; Busch, A.; Gao, Y., Dual subspace discriminative projection learning, Pattern Recognition, 111, Article 107581 pp. (2021) |
| [3] | Cai, H.; Zheng, V. W.; Chang, K. C., A comprehensive survey of graph embedding: Problems, techniques, and applications, IEEE Transactions on Knowledge and Data Engineering, 30, 9, 1616-1637 (2018) |
| [4] | Chen, T.; Kornblith, S.; Norouzi, M.; Hinton, G. E., A simple framework for contrastive learning of visual representations, (Proc. ICML, vol. 119 (2020)), 1597-1607 |
| [5] | Chen, K.; Yao, L.; Zhang, D.; Wang, X.; Chang, X.; Nie, F., A semisupervised recurrent convolutional attention model for human activity recognition, IEEE Transactions on Neural Networks and Learning System, 31, 5, 1747-1756 (2020) |
| [6] | Chuang, C., Robinson, J., Lin, Y., Torralba, A., & Jegelka, S. (2020). Debiased Contrastive Learning. In Proc. NeurIPS. |
| [7] | Dornaika, F.; Khoder, A., Linear embedding by joint robust discriminant analysis and inter-class sparsity, Neural Networks, 127, 141-159 (2020) · Zbl 1472.62093 |
| [8] | Gao, Q.; Xu, S.; Chen, F.; Ding, C.; Gao, X.; Li, Y., \( R_1-2\)-DPCA and face recognition, IEEE Transactions on Cybernetics, 49, 4, 1212-1223 (2019) |
| [9] | Grill, J., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., et al. (2020). Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning. In Proc. NeurIPS. |
| [10] | Han, N.; Wu, J.; Liang, Y.; Fang, X.; Wong, W. K.; Teng, S., Low-rank and sparse embedding for dimensionality reduction, Neural Networks, 108, 202-216 (2018) · Zbl 1434.68413 |
| [11] | He, X.; Cai, D.; Yan, S.; Zhang, H., Neighborhood preserving embedding, (Proc. ICCV (2005)), 1208-1213 |
| [12] | He, K.; Fan, H.; Wu, Y.; Xie, S.; Girshick, R. B., Momentum contrast for unsupervised visual representation learning, (Proc. CVPR (2020)), 9726-9735 |
| [13] | He, X., & Niyogi, P. (2003). Locality Preserving Projections. In Proc. NeurIPS (pp. 153-160). |
| [14] | Huang, H.; Liu, J. M.; Pan, Y. S., Semi-supervised marginal Fisher analysis for hyperspectral image classification, ISPRS, 1-3, 377-382 (2012) |
| [15] | Huang, Z.; Zhu, H.; Zhou, J. T.; Peng, X., Multiple marginal Fisher analysis, IEEE Transactions on Industrial Electronics, 66, 12, 9798-9807 (2019) |
| [16] | Jain, A. K.; Duin, R. P.W.; Mao, J., Statistical pattern recognition: A review, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22, 1, 4-37 (2000) |
| [17] | Kalantidis, Y., Sariyildiz, M. B., Pion, N., Weinzaepfel, P., & Larlus, D. (2020). Hard Negative Mixing for Contrastive Learning. In Proc. NeurIPS. |
| [18] | Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., et al. (2020). Supervised Contrastive Learning. In Proc. NeurIPS. |
| [19] | Kingma, D. P., & Ba, J. (2015). Adam: A Method for Stochastic Optimization. In Proc. ICLR (Poster). |
| [20] | Lai, Z.; Mo, D.; Wong, W. K.; Xu, Y.; Miao, D.; Zhang, D., Robust discriminant regression for feature extraction, IEEE Transactions on Cybernetics, 48, 8, 2472-2484 (2018) |
| [21] | Li, X.; Wang, Q.; Nie, F.; Chen, M., Locality adaptive discriminant analysis framework, IEEE Transactions on Cybernetics, 52, 8, 7291-7302 (2022) |
| [22] | Liao, W.; Pizurica, A.; Scheunders, P.; Philips, W.; Pi, Y., Semi-supervised local discriminant analysis for feature extraction in hyperspectral images, IEEE Transactions on Geoscience and Remote Sensing, 51, 1, 184-198 (2013) |
| [23] | Lu, J.; Lai, Z.; Wang, H.; Chen, Y.; Zhou, J.; Shen, L., Generalized embedding regression: A framework for supervised feature extraction, IEEE Transactions on Neural Networks and Learning System, 33, 1, 185-199 (2022) |
| [24] | Luo, M.; Chang, X.; Nie, L.; Yang, Y.; Hauptmann, A. G.; Zheng, Q., An adaptive semisupervised feature analysis for video semantic recognition, IEEE Transactions on Cybernetics, 48, 2, 648-660 (2018) |
| [25] | Nie, F., Wang, X., & Huang, H. (2014). Clustering and projected clustering with adaptive neighbors. In Proc. KDD (pp. 977-986). |
| [26] | Nie, F.; Wang, Z.; Wang, R.; Li, X., Adaptive local embedding learning for semi-supervised dimensionality reduction, IEEE Transactions on Knowledge and Data Engineering, 1 (2021), early access |
| [27] | Qiao, L.; Chen, S.; Tan, X., Sparsity preserving projections with applications to face recognition, Pattern Recognition, 43, 1, 331-341 (2010) · Zbl 1186.68421 |
| [28] | Ran, R.; Feng, J.; Zhang, S.; Fang, B., A general matrix function dimensionality reduction framework and extension for manifold learning, IEEE Transactions on Cybernetics, 52, 4, 2137-2148 (2022) |
| [29] | Ren, Y.; Wang, Z.; Chen, Y.; Zhao, W., Sparsity preserving discriminant projections with applications to face recognition, Mathematical Problems in Engineering, 2016, 1-12 (2016) · Zbl 1400.94029 |
| [30] | Sugiyama, M., Dimensionality reduction of multimodal labeled data by local Fisher discriminant analysis, Journal of Machine Learning Research, 8, 1027-1061 (2007) · Zbl 1222.68312 |
| [31] | Tian, Y.; Krishnan, D.; Isola, P., Contrastive multiview coding, (Proc. ECCV, vol. 12356 (2020)), 776-794 |
| [32] | Toan, N. T.; Pham, M. T.; Nguyen, T. T.; Huynh, T. T.; Tong, V. V.; Nguyen, Q. V.H., Structural representation learning for network alignment with self-supervised anchor links, Expert Systems with Applications, 165, Article 113857 pp. (2021) |
| [33] | van den Oord, A.; Li, Y.; Vinyals, O., Representation learning with contrastive predictive coding (2018), arXiv preprint, arXiv:1807.03748 |
| [34] | Wang, F., Liu, H., Guo, D., & Sun, F. (2020). Unsupervised Representation Learning by Invariance Propagation. In Proc. NeurIPS. |
| [35] | Wang, A.; Zhao, S.; Liu, J.; Yang, J.; Liu, L.; Chen, G., Locality adaptive preserving projections for linear dimensionality reduction, Expert Systems with Applications, 151, Article 113352 pp. (2020) |
| [36] | Wen, J.; Deng, S.; Fei, L.; Zhang, Z.; Zhang, B.; Zhang, Z., Discriminative regression with adaptive graph diffusion, IEEE Transactions on Neural Networks and Learning System, 1-13 (2022), early access |
| [37] | Yan, S.; Xu, D.; Zhang, B.; Zhang, H.; Yang, Q.; Lin, S., Graph embedding and extensions: A general framework for dimensionality reduction, IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 1, 40-51 (2007) |
| [38] | Yang, W.; Wang, Z.; Sun, C., A collaborative representation based projections method for feature extraction, Pattern Recognition, 48, 1, 20-27 (2015) |
| [39] | Yi, Y.; Wang, J.; Zhou, W.; Fang, Y.; Kong, J.; Lu, Y., Joint graph optimization and projection learning for dimensionality reduction, Pattern Recognition, 92, 258-273 (2019) |
| [40] | Zbontar, J.; Jing, L.; Misra, I.; LeCun, Y.; Deny, S., Barlow twins: Self-supervised learning via redundancy reduction, (Proc. ICML, vol. 139 (2021)), 12310-12320 |
| [41] | Zhang, Z.; Shao, L.; Xu, Y.; Liu, L.; Yang, J., Marginal representation learning with graph structure self-adaptation, IEEE Transactions on Neural Networks and Learning System, 29, 10, 4645-4659 (2018) |
| [42] | Zhang, Y.; Xiang, M.; Yang, B., Low-rank preserving embedding, Pattern Recognition, 70, 112-125 (2017) |
| [43] | Zhang, S., & Yu, G. (2010). Semi-supervised Locality Preserving Projections with Compactness Enhancement. In Proc. int. conf. educ. inf. technol. http://dx.doi.org/10.1109/ICEIT.2010.5607616. |
| [44] | Zhao, W.; Chellappa, R.; Phillips, P. J.; Rosenfeld, A., Face recognition: A literature survey, ACM Computing Surveys, 35, 4, 399-458 (2003) |
| [45] | Zhou, R.; Chang, X.; Shi, L.; Shen, Y.; Yang, Y.; Nie, F., Person reidentification via multi-feature fusion with adaptive graph learning, IEEE Transactions on Neural Networks and Learning System, 31, 5, 1592-1601 (2020) |
| [46] | Zhuge, W.; Hou, C.; Nie, F.; Yi, D., Unsupervised feature extraction using a learned graph with clustering structure, (Proc. ICPR (2016)), 3597-3602 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
