Neurodynamics-driven holistic approaches to semi-supervised feature selection. (English) Zbl 07751979
Summary: Feature selection is a crucial part of machine learning and pattern recognition, which aims at selecting a subset of informative features from the original dataset. Because of label information, supervised feature selection performs better than unsupervised feature selection without label information. However, in the presence of a small number of labeled data and a large number of unlabeled data, it is challenging for supervised feature selection methods to select relevant features. In this paper, we propose three neurodynamics-driven holistic approaches to semi-supervised feature selection via semi-supervised feature redundancy minimization and semi-supervised feature relevancy maximization. We first define information-theoretic semi-supervised similarity coefficient matrix and semi-supervised feature relevancy vector based on multi-information, unsupervised symmetric uncertainty, and entropy to measure feature redundancy and relevancy. We then formulate a fractional programming problem and an iteratively weighted quadratic programming problem based on the semi-supervised similarity coefficient matrix and semi-supervised feature relevancy vector for semi-supervised feature selection. To solve the formulated problems, we delineate three neurodynamic optimization approaches based on two projection neural networks. We elaborate on the experimental results on six benchmark datasets to demonstrate the superior classification performance of the proposed neurodynamic approaches against six existing supervised and semi-supervised feature selection methods.
MSC:
| 68T05 | Learning and adaptive systems in artificial intelligence |
| 94A17 | Measures of information, entropy |
Keywords:
semi-supervised feature selection; information-theoretic measures; neurodynamic optimization; fractional programmingReferences:
| [1] | Ang, J. C.; Mirzal, A.; Haron, H.; Hamed, H. N.A., Supervised, unsupervised, and semi-supervised feature selection: A review on gene selection, IEEE/ACM Transactions on Computational Biology and Bioinformatics, 13, 5, 971-989 (2016) |
| [2] | Azhagusundari, B.; Thanamani, A. S., Feature selection based on information gain, International Journal of Innovative Technology and Exploring Engineering, 2, 2, 18-21 (2013) |
| [3] | Battiti, R., Using mutual information for selecting features in supervised neural net learning, IEEE Transactions on Neural Networks, 5, 4, 537-550 (1994) |
| [4] | Bian, W.; Ma, L.; Qin, S.; Xue, X., Neural network for nonsmooth pseudoconvex optimization with general convex constraints, Neural Networks, 101, 1-14 (2018) · Zbl 1443.90267 |
| [5] | Blum, A.; Langley, P., Selection of relevant features and examples in machine learning, Artificial Intelligence, 97, 1, 245-271 (1997) · Zbl 0904.68142 |
| [6] | Boyd, S.; Parikh, N.; Chu, E.; Peleato, B.; Eckstein, J., Distributed optimization and statistical learning via the alternating direction method of multipliers, Foundations & Trends in Machine Learning, 3, 1, 1-122 (2010) · Zbl 1229.90122 |
| [7] | Breiman, L., Random forests, Machine Learning, 45, 5-32 (2001) · Zbl 1007.68152 |
| [8] | Chakraborty, D.; Pal, N. R., Selecting useful groups of features in a connectionist framework, IEEE Transactions on Neural Networks, 19, 3, 381-396 (2008) |
| [9] | Chakraborty, R.; Pal, N. R., Feature selection using a neural framework with controlled redundancy, IEEE Transactions on Neural Networks and Learning Systems, 26, 1, 35-50 (2014) |
| [10] | Che, H.; Wang, J., A two-timescale duplex neurodynamic approach to biconvex optimization, IEEE Transactions on Neural Networks and Learning Systems, 30, 8, 2503-2514 (2019) |
| [11] | Chung, I. F.; Chen, Y. C.; Pal, N., Feature selection with controlled redundancy in a fuzzy rule based framework, IEEE Transactions on Fuzzy Systems, 26, 2, 734-748 (2018) |
| [12] | Cover, T. M.; Thomas, J. A., Elements of information theory (2006), John Wiley & Sons · Zbl 1140.94001 |
| [13] | Di Marco, M.; Forti, M.; Pancioni, L.; Innocenti, G.; Tesi, A., Memristor neural networks for linear and quadratic programming problems, IEEE Transactions on Cybernetics, 52, 3, 1822-1835 (2022) |
| [14] | Duda, R. O.; Hart, P. E., Pattern classification and scene analysis (1973), Wiley New York · Zbl 0277.68056 |
| [15] | Forti, M.; Nistri, P.; Quincampoix, M., Convergence of neural networks for programming problems via a nonsmooth Lojasiewicz inequality, IEEE Transactions on Neural Networks, 17, 6, 1471-1486 (2006) |
| [16] | Freeman, C.; Kulić, D.; Basir, O., Feature-selected tree-based classification, IEEE Transactions on Cybernetics, 43, 6, 1990-2004 (2013) |
| [17] | Gui, J.; Sun, Z.; Ji, S.; Tao, D.; Tan, T., Feature selection based on structured sparsity: A comprehensive study, IEEE Transactions on Neural Networks and Learning Systems, 28, 7, 1490-1507 (2016) |
| [18] | Guo, Z.; Liu, Q.; Wang, J., A one-layer recurrent neural network for pseudoconvex optimization subject to linear equality constraints, IEEE Transactions on Neural Networks, 22, 12, 1892-1900 (2011) |
| [19] | Hopfield, J. J.; Tank, D. W., Computing with neural circuits - a model, Science, 233, 4764, 625-633 (1986) |
| [20] | Hunter, D. R.; Lange, K., A tutorial on MM algorithms, The American Statistician, 58, 1, 30-37 (2004) |
| [21] | Jiang, B.; Li, C.; De Rijke, M.; Yao, X.; Chen, H., Probabilistic feature selection and classification vector machine, ACM Transactions on Knowledge Discovery from Data, 13, 2, 21-27 (2019) |
| [22] | Kohavi, R.; John, G. H., Wrappers for feature subset selection, Artificial Intelligence, 97, 1, 273-324 (1997) · Zbl 0904.68143 |
| [23] | Koller, D., & M., S. (1996). Toward Optimal Feature Selection. In Proceedings of the thireteenth interational conference on machine learning (pp. 284-292). |
| [24] | Li, J.; Cheng, K.; Wang, S.; Morstatter, F.; Trevino, R. P.; Tang, J.; Liu, H., Feature selection: A data perspective, ACM Computing Surveys, 50, 6, 1-45 (2017) |
| [25] | Li, X.; Wang, Y.; Ruiz, R., A survey on sparse learning models for feature selection, IEEE Transactions on Cybernetics, 52, 3, 1642-1660 (2022) |
| [26] | Lin, D.; Tang, X., Conditional infomax learning: an integrated framework for feature extraction and fusion, (European conference on computer vision (2006)), 68-82 |
| [27] | Liu, S.; Jiang, H.; Zhang, L.; Mei, X., A neurodynamic optimization approach for complex-variables programming problem, Neural Networks, 129, 280-287 (2020) · Zbl 1475.90058 |
| [28] | Liu, N.; Qin, S., A neurodynamic approach to nonlinear optimization problems with affine equality and convex inequality constraints, Neural Networks, 109, 147-158 (2019) · Zbl 1472.90128 |
| [29] | Liu, Q.; Wang, J., A one-layer projection neural network for nonsmooth optimization subject to linear equalities and bound constraints, IEEE Transactions on Neural Networks and Learning Systems, 24, 5, 812-824 (2013) |
| [30] | Liu, N.; Wang, J.; Qin, S., A one-layer recurrent neural network for nonsmooth pseudoconvex optimization with quasiconvex inequality and affine equality constraints, Neural Networks, 147, 1-9 (2022) · Zbl 07749919 |
| [31] | Nie, F.; Wang, Z.; Tian, L.; Wang, R.; Li, X., Subspace sparse discriminative feature selection, IEEE Transactions on Cybernetics, 52, 6, 4221-4233 (2022) |
| [32] | Nie, F.; Yang, S.; Zhang, R.; Li, X., A general framework for auto-weighted feature selection via global redundancy minimization, IEEE Transactions on Image Processing, 28, 5, 2428-2438 (2019) · Zbl 1411.94009 |
| [33] | Peng, H.; Long, F.; Ding, C., Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy, IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 8, 1226-1238 (2005) |
| [34] | Rodriguezlujan, I.; Huerta, R.; Elkan, C.; Cruz, C. S., Quadratic programming feature selection, Journal of Machine Learning Research, 11, 1491-1516 (2010) · Zbl 1242.68245 |
| [35] | Sheikhpour, R.; Sarram, M. A.; Gharaghani, S.; Chahooki, M. A.Z., A survey on semi-supervised feature selection methods, Pattern Recognition, 64, 141-158 (2017) · Zbl 1429.68239 |
| [36] | Sun, Y.; Babu, P.; Palomar, D. P., Majorization-minimization algorithms in signal processing, communications, and machine learning, IEEE Transactions on Signal Processing, 65, 3, 794-816 (2016) · Zbl 1414.94595 |
| [37] | Wang, J., Analysis and design of a \(k\)-winners-take-all model with a single state variable and the Heaviside step activation function, IEEE Transactions on Neural Networks, 21, 9, 1496-1506 (2010) |
| [38] | Wang, Y.; Li, X.; Ruiz, R., Weighted general group lasso for gene selection in cancer classification, IEEE Transactions on Cybernetics, 49, 8, 2860-2873 (2019) |
| [39] | Wang, Y.; Li, X.; Wang, J., A neurodynamic optimization approach to supervised feature selection via fractional programming, Neural Networks, 136, 194-206 (2021) · Zbl 1521.68167 |
| [40] | Wang, D.; Nie, F.; Huang, H., Feature selection via global redundancy minimization, IEEE Transactions on Knowledge and Data Engineering, 27, 10, 2743-2755 (2015) |
| [41] | Wang, Y.; Wang, J.; Che, H., Two-timescale neurodynamic approaches to supervised feature selection based on alternative problem formulations, Neural Networks, 142, 180-191 (2021) |
| [42] | Wang, Y.; Wang, J.; Liao, H.; Chen, H., An efficient semi-supervised representatives feature selection algorithm based on information theory, Pattern Recognition, 61, 511-523 (2017) · Zbl 1428.68262 |
| [43] | Wang, Y.; Zhang, Z.; Lin, Y., Multi-cluster feature selection based on isometric mapping, IEEE/CAA Journal of Automatica Sinica, 9, 3, 570-572 (2022) |
| [44] | Xia, Y.; Leung, H.; Wang, J., A projection neural network and its application to constrained optimization problems, IEEE Transactions on Circuits and Systems: Part I, 49, 4, 447-458 (2002) · Zbl 1368.92019 |
| [45] | Xu, J., Adapt the mRMR criterion for unsupervised feature selection, (Advanced data mining and applications - 6th international conference, ADMA 2010, Chongqing, China, November 19-21, 2010, proceedings, part II (2010)), 111-121 |
| [46] | Xu, C.; Chai, Y.; Qin, S.; Wang, Z.; Feng, J., A neurodynamic approach to nonsmooth constrained pseudoconvex optimization problem, Neural Networks, 124, 180-192 (2020) · Zbl 1444.90113 |
| [47] | Yang, M.; Chen, Y. J.; Ji, G. L., Semi \(\text{\_}\) Fisher score: A semi-supervised method for feature selection, (International conference on machine learning and cybernetics (2010)), 527-532 |
| [48] | Yang, H., & Moody, J. (1999). Feature selection based on joint mutual information. In Proc. of international ICSC symposium on advances in intelligent data analysis (pp. 22-25). |
| [49] | Yeung, R., A new outlook on Shannon’s information measures, IEEE Transactions on Information Theory, 37, 3, 466-474 (1991) · Zbl 0741.94009 |
| [50] | Zhang, P.; Gao, W.; Hu, J.; Li, Y., A conditional-weight joint relevance metric for feature relevancy term, Engineering Applications of Artificial Intelligence, 106, Article 104481 pp. (2021) |
| [51] | Zhang, H.; Wang, J.; Sun, Z.; Zurada, J. M.; Pal, N. R., Feature selection for neural networks using group lasso regularization, IEEE Transactions on Knowledge and Data Engineering, 32, 4, 659-673 (2020) |
| [52] | Zhao, J.; Ke, L.; He, X., Locality sensitive semi-supervised feature selection, Neurocomputing, 71, 10-12, 1842-1849 (2008) |
| [53] | Zhao, H.; Li, Q.; Wang, Z.; Nie, F., Joint adaptive graph learning and discriminative analysis for unsupervised feature selection, Cognitive Computation, 14, 3, 1211-1221 (2022) |
| [54] | Zhao, Y.; Liao, X.; He, X., Novel projection neurodynamic approaches for constrained convex optimization, Neural Networks, 150, 336-349 (2022) · Zbl 1533.90081 |
| [55] | Zhao, Y.; Liu, Q., A consensus algorithm based on collective neurodynamic system for distributed optimization with linear and bound constraints, Neural Networks, 122, 144-151 (2019) · Zbl 1440.90050 |
| [56] | Zheng, Z.; Liu, H., Semi-supervised feature selection via spectral analysis, (SIAM international conference on data mining (2007)), 641-646 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
