Membership-margin based feature selection for mixed type and high-dimensional data: theory and applications. (English) Zbl 1390.68531
Summary: The present paper describes a new feature weighting method based on a membership margin. Distinctive properties of the proposed method include its capability to process problems characterized by mixed-type data (quantitative, qualitative and interval) as well as a huge number of features. The key idea is to map simultaneously all the features of different types into a common space; the membership space. Once all features are represented in a homogeneous space, a feature weighting task can be performed in unified way. This weighting approach is integrated here within a fuzzy classifier through a fuzzy rule weighted concept in order to improve its performance. Each antecedent fuzzy set in the fuzzy if-then rule is weighted to characterize the importance of each proposition and therefore its corresponding feature. Weight estimation process is based on membership margin maximization to estimate a fuzzy weight of each feature in the membership space. Experiments on low and high dimensional real-world datasets demonstrate that the proposed approach can improve significantly the performance of the fuzzy rule-based as well as other state of the art classifiers and can even outperform classical feature weighting approaches. In particular, we show that this approach can yield meaningful results on two real-world applications for cancer prognosis and industrial process diagnosis.
MSC:
| 68T05 | Learning and adaptive systems in artificial intelligence |
| 62H30 | Classification and discrimination; cluster analysis (statistical aspects) |
References:
| [1] | Abe, S.; Thawonmas, R., A fuzzy classifier with ellipsoidal regions, IEEE Trans. Fuzzy Syst., 5, 358-368 (1997) |
| [3] | Aguilar-Martin, J.; Lopez de Mantaras, R., The process of classification and learning the meaning of linguistic descriptors of concepts, Approx. Reason. Dec. Anal., 165-175 (1982) |
| [4] | Alcalá-Fdez, J.; Alcalá, R.; Herrera, F., A fuzzy association rule-based classification model for high-dimensional problems with genetic rule selection and lateral tuning, IEEE Trans. Fuzzy Syst., 19, 857-872 (2011) |
| [5] | Baldi, P.; Bunak, S., Bioinformatics: The Machine Learning Approach (2001), MIT Press: MIT Press Cambridge · Zbl 0992.92024 |
| [6] | Banerjee, M.; Pal, N. R., Feature selection with SVD entropy: some modification and extension, Inf. Sci. (NY), 264, 118-134 (2014) · Zbl 1335.68194 |
| [7] | Bezdek, J. C., Pattern Recognition with Fuzzy Objective Function Algorithms (1981), Kluwer Academic Publishers: Kluwer Academic Publishers Norwell, Boston, MA · Zbl 0503.68069 |
| [8] | Bhattacharjee, A.; Richards, W. G.; Staunton, J.; Li, C.; Monti, S.; Vasa, P., Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma subclasses, Proc. Natl. Acad. Sci. USA, 98, 13790-13795 (2001) |
| [11] | Chen, S.-M., A weighted fuzzy reasoning algorithm for medical diagnosis, Dec. Supp. Syst., 11, 37-43 (1994) |
| [12] | Chiu, S. L., Extracting fuzzy rules from data for function approximation and pattern classification, (Dubois, D.; Prade, H.; Yager, R., Fuzzy Information Engineering: A Guided Tour of Applications (1997), John Wiley & Sons), (chapter 9) |
| [13] | Cover, T.; Hart, P., Nearest neighbor pattern classification, IEEE Trans. Inf. Theory, 13 (1967) · Zbl 0154.44505 |
| [14] | Cross, V. V.; Sudkamp, T. A., Similarity and Compatibility in Fuzzy Set Theory - Assessment and Applications (2002), Physica-Verlag · Zbl 0992.03066 |
| [16] | Dietterich Thomas, G., Machine-learning research: four current directions, AI Mag., 18, 97-136 (1997) |
| [17] | Dubois, D.; Prade, H., The three semantics of fuzzy sets, Fuzzy Sets Syst., 90, 141-150 (1997) · Zbl 0919.04006 |
| [18] | Dubois, D.; Prade, H.; Testemale, C., Weighted fuzzy pattern matching, Fuzzy Sets Syst., 28, 313-331 (1988) · Zbl 0658.94026 |
| [19] | Freund, Y.; Schapire, R. E., A decision-theoretic generalization of on-line learning and an application to boosting, J. Comput. Syst. Sci., 55, 119-139 (1997) · Zbl 0880.68103 |
| [20] | Gevaert, O.; De Smet, F.; Timmerman, D.; Moreau, Y.; De Moor, B., Predicting the prognosis of breast cancer by integrating clinical and microarray data with Bayesian networks, Bioinformatics, 22 (2006) |
| [21] | Gheyas, I. A.; Smith, L. S., Feature subset selection in large dimensionality domains, Pattern Recog., 43, 5-13 (2010) · Zbl 1192.68574 |
| [23] | Golub, T. R.; Slonim, D. K.; Tamayo, P.; Huard, C.; Gaasenbeek, M.; Mesirov, J., Molecular classification of cancer: class discovery and class prediction by gene expression monitoring, Science, 286, 531-537 (1999) |
| [24] | González, A.; Pérez, R., Selection of relevant features in a fuzzy genetic learning algorithm, IEEE Trans. Syst. Man, Cybern. Part B Cybern., 31, 417-425 (2001) |
| [25] | Gowda, K. C.; Diday, E., Symbolic clustering using a new similarity measure, IEEE Trans. Syst. Man. Cybern., 22 (1992) |
| [27] | Guyon, I.; Elisseeff, A., An introduction to variable and feature selection, J. Mach. Learn. Res., 3, 1157-1182 (2003) · Zbl 1102.68556 |
| [28] | Guyon, I.; Weston, J.; Barnhill, S.; Vapnik, V., Gene selection for cancer classification using support vector machines, Mach. Learn., 46, 389-422 (2002) · Zbl 0998.68111 |
| [29] | Han, X.; Chang, X.; Quan, L.; Xiong, X.; Li, J.; Zhang, Z., Feature subset selection by gravitational search algorithm optimization, Inf. Sci. (NY), 281, 128-146 (2014) |
| [30] | Hedjazi, L.; Aguilar-Martin, J.; Le Lann, M.-V., Similarity-margin based feature selection for symbolic interval data, Pattern Recog. Lett., 32, 578-585 (2011) |
| [32] | Hedjazi, L.; Le Lann, M.-V.; Kempowsky, T.; Dalenc, F.; Aguilar-Martin, J.; Favre, G., Symbolic data analysis to defy low signal-to-noise ratio in microarray data for breast cancer prognosis, J. Comput. Biol., 20, 610-620 (2013) |
| [33] | Hsu, H. H.; Hsieh, C. W.; Da Lu, M., Hybrid feature selection by combining filters and wrappers, Expert Syst. Appl., 38, 8144-8150 (2011) |
| [34] | Isaza, C.; Kempowsky, T.; Aguilar-Martin, J.; Gauthier, A., Qualitative data classification using LAMDA and other soft-computer methods, (Recent Advances in Artificial Intelligence Research and Development (2004), IOS Press) |
| [35] | Ishibuchi, H.; Nakashima, T., Effect of rule weights in fuzzy rule-based classification systems, IEEE Trans. Fuzzy Syst., 9, 506-515 (2001) |
| [36] | Ishibuchi, H.; Nozaki, K.; Tanaka, H., Distributed representation of fuzzy rules and its application to pattern classification, Fuzzy Sets Syst., 52, 21-32 (1992) |
| [37] | Ishibuchi, H.; Yamamoto, T., Rule weight specification in fuzzy rule-based classification systems, IEEE Trans. Fuzzy Syst., 13, 428-435 (2005) |
| [38] | Jaccard, P., Étude comparative de la distribution florale dans une portion des Alpes et des Jura, Bull. La Société Vaudoise Des Sci. Nat., 37, 547-579 (1901) |
| [39] | Jahromi, M. Z.; Taheri, M., A proposed method for learning rule weights in fuzzy rule-based classification systems, Fuzzy Sets Syst., 159, 449-459 (2008) · Zbl 1176.68161 |
| [40] | Jain, A. K.; Murty, M. N.; Flynn, P. J., Data clustering: a review, ACM Comput. Surv., 31, 264-323 (1999) |
| [41] | Jensen, R.; Shen, Q., Fuzzy-rough sets assisted attribute selection, IEEE Trans. Fuzzy Syst., 15, 73-89 (2007) |
| [42] | Jenssen, T.; Hovig, E., Gene-expression profiling in breast cancer, Lancet, 365, 634-635 (2005) |
| [44] | Kempowsky, T.; Valle, C.; Lacroix-Triki, M.; Hedjazi, L.; Trouilh, L.; Lamarre, S., Fuzzy logic selection as a new reliable tool to identify molecular grade signatures in breast cancer - the INNODIAG study, BMC Med. Genomics, 8, 3 (2015) |
| [45] | Khan, J.; Wei, J. S.; Ringnér, M.; Saal, L. H.; Ladanyi, M.; Westermann, F., Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks, Nat. Med., 7, 673-679 (2001) |
| [48] | Kohavi, R., Wrappers for feature subset selection, Artif. Intell., 97, 273-324 (1997) · Zbl 0904.68143 |
| [50] | Kuncheva, L. I., Fuzzy rough sets: application to feature selection, Fuzzy Sets Syst., 51, 147-153 (1992) |
| [51] | Lafferty, J.; Wasserman, L., Challenges in statistical machine learning, Stat. Sin., 16, 307-322 (2006) |
| [52] | Lee, C. C., Fuzzy logic in control systems: fuzzy logic controller - part I, IEEE Trans. Syst. Man Cybern., 20, 404-418 (1990) · Zbl 0707.93036 |
| [53] | Lee, H. M.; Chen, C. M.; Chen, J. M.; Jou, Y. L., An efficient fuzzy classifier with feature selection based on fuzzy entropy, IEEE Trans. Syst. Man. Cybern. B. Cybern., 31, 426-432 (2001) |
| [54] | Li, Y.; Lu, B.-L., Feature selection based on loss-margin of nearest neighbor classification, Pattern Recog., 42, 1914-1921 (2009) · Zbl 1192.68585 |
| [55] | Li, Y.; Wu, Z. F., Fuzzy feature selection based on min-max learning rule and extension matrix, Pattern Recog., 41, 217-226 (2008) · Zbl 1122.68539 |
| [56] | Lu, W.; Zhang, L.; Liu, X.; Yang, J.; Pedrycz, W., A human-computer cooperation fuzzy C-means clustering with interval-valued weights, Int. J. Intell. Syst., 30, 81-98 (2015) |
| [57] | Marcelloni, F., Feature selection based on a modified fuzzy C-means algorithm with supervision, Inf. Sci. (NY), 151, 201-226 (2003) · Zbl 1012.68667 |
| [58] | Medasani, S.; Kim, J.; Krishnapuram, R., An overview of membership function generation techniques for pattern recognition, Int. J. Approx. Reason., 19, 391-417 (1998) · Zbl 0947.68555 |
| [59] | Meiri, R.; Zahavi, J., Using simulated annealing to optimize the feature selection problem in marketing applications, Eur. J. Oper. Res., 171, 842-858 (2006) · Zbl 1116.90069 |
| [60] | Mitra, P.; Murthy, C. A.; Pal, S. K., Unsupervised feature selection using feature similarity, IEEE Trans. Pattern Anal. Mach. Intell., 24, 301-312 (2002) |
| [61] | Moaref, A.; Naeini, V. S., A fuzzy-rough approach for finding various minimal data reductions using ant colony optimization, J. Intell. Fuzzy Syst., 26 (2014) · Zbl 1305.68178 |
| [62] | Pedrycz, W., Fuzzy Sets Engineering (1995), CRC press: CRC press Boca Raton, FL · Zbl 0925.93507 |
| [63] | Pedrycz, W.; Gomide, F., Fuzzy Systems Engineering (2007), John Wiley & Sons: John Wiley & Sons Hoboken, NJ, USA |
| [64] | Ramaswamy, S.; Tamayo, P.; Rifkin, R.; Mukherjee, S.; Yeang, C. H.; Angelo, M., Multiclass cancer diagnosis using tumor gene expression signatures, Proc. Natl. Acad. Sci. USA, 98, 15149-15154 (2001) |
| [65] | Ramze Rezaee, M.; Goedhart, B.; Lelieveldt, B. P.F.; Reiber, J. H.C., Fuzzy feature selection, Pattern Recog., 32, 2011-2019 (1999) |
| [66] | Robnik-Šikonja, M.; Kononenko, I., Theoretical and empirical analysis of ReliefF and RReliefF, Mach. Learn., 53, 23-69 (2003) · Zbl 1076.68065 |
| [67] | Sanz, J. A.; Fernández, A.; Bustince, H.; Herrera, F., Improving the performance of fuzzy rule-based classification systems with interval-valued fuzzy sets and genetic amplitude tuning, Inf. Sci. (NY), 180, 3674-3685 (2010) |
| [68] | Sanz, J. A.; Fernández, A.; Bustince, H.; Herrera, F., A genetic tuning to improve the performance of fuzzy rule-based classification systems with interval-valued fuzzy sets: degree of ignorance and lateral position, Int. J. Approx. Reason., 52, 751-766 (2011) |
| [69] | Schmitt, E.; Bombardier, V.; Wendling, L., Improving fuzzy rule classifier by extracting suitable features from capacities with respect to the Choquet integral, IEEE Trans. Syst. Man, Cybern. Part B Cybern., 38, 1195-1206 (2008) |
| [71] | Singh, D.; Febbo, P. G.; Ross, K.; Jackson, D. G.; Manola, J.; Ladd, C., Gene expression correlates of clinical prostate cancer behavior, Cancer Cell, 1, 203-209 (2002) |
| [72] | Straver, M. E.; Glas, A. M.; Hannemann, J.; Wesseling, J.; van De Vijver, M. J.; Rutgers, E. J.T., The 70-gene signature as a response predictor for neoadjuvant chemotherapy in breast cancer, Breast Cancer Res. Treat., 119, 551-558 (2010) |
| [73] | Subramaniam, D. S.; Isaacs, C., Utilizing prognostic and predictive factors in breast cancer, Curr. Treat Options Oncol., 6, 147-159 (2005) |
| [74] | Sugeno, M., An introductory survey of fuzzy control, Inf. Sci. (NY), 36, 59-83 (1985) · Zbl 0586.93053 |
| [75] | Sun, Y., Iterative RELIEF for feature weighting: algorithms, theories, and applications, IEEE Trans. Pattern Anal. Mach. Intell., 29, 1035-1051 (2007) |
| [76] | Sun, Y.; Goodison, S.; Li, J.; Liu, L.; Farmerie, W., Improved breast cancer prognosis through the combination of clinical and genetic markers, Bioinformatics, 23, 30-37 (2007) |
| [78] | van’t Veer, L. J.; Dai, H.; van De Vijver, M. J.; He, Y. D.; Hart, A.aM.; Mao, M., Gene expression profiling predicts clinical outcome of breast cancer, Nature, 415, 530-536 (2002) |
| [79] | Vapnik, V., Statistical Learning Theory (1998), John Wiley & Sons · Zbl 0935.62007 |
| [80] | Vieira, S. M.; Sousa, J. M.C.; Kaymak, U., Fuzzy criteria for feature selection, Fuzzy Sets Syst., 189, 1-18 (2012) |
| [81] | van De Vijver, M. J.; He, Y. D.; van’t Veer, L. J.; Dai, H.; Hart, A. A.M.; Voskuil, D. W., A gene-expression signature as a predictor of survival in breast cancer, N. Engl. J. Med., 347, 1999-2009 (2002) |
| [82] | Wang, S.; Pedrycz, W.; Zhu, Q.; Zhu, W., Unsupervised feature selection via maximum projection and minimum redundancy, Knowl.-Based Syst., 75, 19-29 (2015) |
| [83] | Wang, X.; Wang, Y.; Wang, L., Improving fuzzy C-means clustering based on feature-weight learning, Pattern Recog. Lett., 25, 1123-1132 (2004) |
| [84] | Wang, Z.; Li, M.; Li, J., A multi-objective evolutionary algorithm for feature selection based on mutual information with a new redundancy measure, Inf. Sci. (NY), 307, 73-88 (2015) · Zbl 1387.68200 |
| [87] | Wettschereck, D.; Aha, D. W.; Mohri, T., A review and empirical evaluation of feature weighting methods for a class of lazy learning algorithms, Learning, 1-37 (1997) |
| [88] | Wygralak, M., An axiomatic approach to scalar cardinalities of fuzzy sets, Fuzzy Sets Syst., 110, 175-179 (2000) · Zbl 0945.03080 |
| [89] | Zadeh, L. A., Fuzzy sets, Inf. Control, 8, 338-353 (1965) · Zbl 0139.24606 |
| [90] | Zhang, L.; Zhang, L.; Pedrycz, W.; Lu, W.; Liu, X., An interval weighed fuzzy C-means clustering by genetically guided alternating optimization, Expert Syst. Appl., 41, 5960-5971 (2014) |
| [91] | Zwick, R.; Carlstein, E.; Budescu, D. V., Measures of similarity among fuzzy concepts: a comparative analysis, Int. J. Approx. Reason., 1, 221-242 (1987) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
