Feature selection in mixed data: a method using a novel fuzzy rough set-based information entropy. (English) Zbl 1412.68198
Summary: Feature selection in the data with different types of feature values, i.e., the heterogeneous or mixed data, is especially of practical importance because such types of data sets widely exist in real world. The key issue for feature selection in mixed data is how to properly deal with different types of the features or attributes in the data set. Motivated by the fuzzy rough set theory which allows different fuzzy relations to be defined for different types of attributes to measure the similarity between objects and in view of the effectiveness of entropy to measure information uncertainty, we propose in this paper a fuzzy rough set-based information entropy for feature selection in a mixed data set. It is proved that the newly-defined entropy meets the common requirement of monotonicity and can equivalently characterize the existing attribute reductions in the fuzzy rough set theory. Then, a feature selection algorithm is formulated based on the proposed entropy and a filter-wrapper method is suggested to select the best feature subset in terms of classification accuracy. An extensive numerical experiment is further conducted to assess the performance of the feature selection method and the results are satisfactory.
MSC:
| 68T05 | Learning and adaptive systems in artificial intelligence |
| 68T37 | Reasoning under uncertainty in the context of artificial intelligence |
References:
| [1] | Bhatt, R. B.; Gopal, M., On fuzzy-rough sets approach to feature selection, Pattern Recognit. Lett., 26, 965-975 (2005) |
| [2] | Boixader, D.; Jacas, J.; Recasens, J., Upper and lower approximations of fuzzy sets, Int. J. Gen. Syst., 29, 555-568 (2000) · Zbl 0955.03056 |
| [3] | Chen, D. G., Theory and Methods of Fuzzy Rough Sets (2013), Science Press: Science Press Beijing |
| [4] | Chen, D. G.; Hu, Q. H.; Yang, Y. P., Parameterized attribute reduction with Gaussian kernel based fuzzy rough sets, Inf. Sci., 181, 5169-5179 (2011) · Zbl 1239.68059 |
| [5] | Chen, D. G.; Yang, Y. Y., Attribute reduction for heterogeneous data based on combination of classical and fuzzy rough set models, IEEE Trans. Fuzzy Syst., 22, 1325-1334 (2014) |
| [6] | Ching, J. Y.; Wong, A. K.C.; Chan, K. C.C., Class-dependent discretization for inductive learning from continuous and mixed-mode data, IEEE Trans. Pattern Anal. Mach. Intell., 17, 641-651 (1995) |
| [7] | Chmielewski, M. R.; Grzymala-Busse, J. W., Global discretization of continuous attributes as preprocessing for machine learning, Int. J. Approx. Reason., 15, 319-331 (1996) · Zbl 0949.68560 |
| [8] | Cornelis, C.; Jensen, R.; Hurtado, G.; Ślȩzak, D., Attribute selection with fuzzy decision reducts, Inf. Sci., 180, 209-224 (2010) · Zbl 1183.68604 |
| [9] | Dai, J. H.; Tian, H. W., Fuzzy rough set model for set-valued data, Fuzzy Sets Syst., 229, 54-68 (2013) · Zbl 1284.68536 |
| [10] | Dash, M.; Liu, H., Feature selection for classification, Intell. Data Anal., 1, 131-156 (1997) |
| [11] | Dash, M.; Liu, H., Consistency-based search in feature selection, Artif. Intell., 151, 155-176 (2003) · Zbl 1082.68791 |
| [12] | J. Dougherty, R. Kohavi, M. Sahami, Supervised and unsupervised discretization of continuous features, In: Proceedings of the 12th International Conference on Machine Learning, 1995, pp. 194-202.; J. Dougherty, R. Kohavi, M. Sahami, Supervised and unsupervised discretization of continuous features, In: Proceedings of the 12th International Conference on Machine Learning, 1995, pp. 194-202. |
| [13] | Dubois, D.; Prade, H., Rough fuzzy sets and fuzzy rough sets, Int. J. Gen. Syst., 17, 191-209 (1990) · Zbl 0715.04006 |
| [14] | Fu, X. H.; Yang, K.; Huang, J. Z.; Cui, L. Z., Dynamic non-parametric joint sentiment topic mixture model, Knowl.-Based Syst., 82, 102-114 (2015) |
| [15] | Gan, J. Q.; Hasan, B. A.S.; Tsui, C. S.L., A filter-dominating hybrid sequential forward floating search method for feature subset selection in high-dimensional space, Int. J. Mach. Learn. Cybern., 5, 413-423 (2014) |
| [16] | Hu, Q. H.; Xie, Z. X.; Yu, D. R., Comments on fuzzy probabilistic approximations spaces and their information measures, IEEE Trans. Fuzzy Syst., 16, 549-551 (2008) |
| [17] | Hu, Q. H.; Yu, D. R.; Liu, J. F.; Wu, C. X., Neighborhood rough set based heterogeneous feature subset selection, Inf. Sci., 178, 3577-3594 (2008) · Zbl 1154.68466 |
| [18] | Hu, Q. H.; Yu, D. R.; Xie, Z. X.; Liu, J. F., Fuzzy probabilistic approximations spaces and their information measures, IEEE Trans. Fuzzy Syst., 14, 191-201 (2006) |
| [19] | Hu, Q. H.; Yu, D. R.; Xie, Z. X., Information-preserving hybrid data reduction based on fuzzy-rough techniques, Pattern Recognit. Lett., 27, 414-423 (2006) |
| [20] | Hu, Q. H.; Zhang, L.; Chen, D. G.; Pedrycz, W.; Yu, D. R., Gaussian kernel based fuzzy rough setsmodel, uncertainty measures and applications, Int. J. Approx. Reason., 51, 453-471 (2010) · Zbl 1205.68424 |
| [21] | Jensen, R.; Shen, Q., Fuzzy-rough attribute reduction with application to web categorization, Fuzzy Sets Syst., 141, 469-485 (2004) · Zbl 1069.68609 |
| [22] | Jensen, R.; Shen, Q., Fuzzy-rough sets assisted attribute selection, IEEE Trans. Fuzzy Syst., 15, 73-89 (2007) |
| [23] | Jensen, R.; Shen, Q., New approaches to fuzzy-rough feature selection, IEEE Trans. Fuzzy Syst., 17, 824-838 (2009) |
| [24] | Kohavi, R.; John, G. H., Wrappers for feature subset selection, Artif. Intell., 97, 273-324 (1997) · Zbl 0904.68143 |
| [25] | Li, J. K.; Zhao, H.; Zhu, W., Fast randomized algorithm with restart strategy for minimal test cost feature selection, Int. J. Mach. Learn. Cybern., 6, 435-442 (2015) |
| [26] | Liang, J. Y.; Zhao, X. W.; Li, D. Y.; Cao, F. Y.; Dang, C. Y., Determining the number of clusters using information entropy for mixed data, Pattern Recognit., 45, 2251-2265 (2012) · Zbl 1234.68343 |
| [27] | Liu, S.; Bai, X., Discriminative features for image classification and retrieval, Pattern Recognit. Lett., 33, 744-751 (2012) |
| [28] | Liu, X. D.; Pedrycz, W.; Chai, T. Y.; Song, M. L., The development of fuzzy rough sets with the use of structures and algebras of axiomatic fuzzy sets, IEEE Trans. Knowl. Data Eng., 21, 443-462 (2009) |
| [29] | Lu, S. X.; Wang, X. Z.; Zhang, G. Q.; Zhou, X., Effective algorithms of the Moore-Penrose inverse matrices for extreme learning machine, Intell. Data Anal., 19, 743-760 (2015) |
| [30] | Mi, J. S.; Leung, Y.; Wu, W. Z., An uncertainty measure in partition-based fuzzy rough sets, Int. J. Gen. Syst., 34, 77-90 (2005) · Zbl 1125.03309 |
| [31] | Mi, J. S.; Leung, Y.; Zhao, H. Y.; Feng, T., Generalized fuzzy rough sets determined by a triangular norm, Inf. Sci., 178, 3203-3213 (2008) · Zbl 1151.03344 |
| [32] | Mi, J. S.; Zhang, W. X., An axiomatic characterization of a fuzzy generalization of rough sets, Inf. Sci., 160, 235-249 (2004) · Zbl 1041.03038 |
| [33] | Mitov, I.; Ivanova, K.; Markov, K.; Velychko, V.; Stanchev, P.; Vanhoof, K., Comparison of discretization methods for preprocessing data for pyramidal growing network classification method, New Trends Intell. Technol., 31-39 (2009) |
| [34] | Mitra, P.; Murthy, C. A.; Pal, S. K., Unsupervised feature selection using feature similarity, IEEE Trans. Pattern Anal. Mach. Intell., 24, 301-312 (2002) |
| [35] | Morsi, N. N.; Yakout, M. M., Axiomatics for fuzzy rough sets, Fuzzy Sets Syst., 100, 327-342 (1998) · Zbl 0938.03085 |
| [36] | Pawlak, Z., Rough sets, Int. J. Comput. Inf. Sci., 11, 341-356 (1982) · Zbl 0501.68053 |
| [37] | Peng, H. C.; Long, F. H.; Ding, C., Feature selection based on mutual informationcriteria of max-dependency, max-relevance, and min-redundancy, IEEE Trans. Pattern Anal. Mach. Intell., 27, 1226-1238 (2005) |
| [38] | Qian, Y. H.; Wang, Q.; Cheng, H. H.; Liang, J. Y.; Dang, C. Y., Fuzzy-rough feature selection accelerator, Fuzzy Sets and Systems, 258, 61-78 (2015) · Zbl 1335.68271 |
| [39] | Quinlan, J. R., Induction of decision trees, Mach. Learn., 1, 81-106 (1986) |
| [40] | Radzikowska, A. M.; Kerre, E. E., A comparative study of fuzzy rough sets, Fuzzy Sets Syst., 126, 137-155 (2002) · Zbl 1004.03043 |
| [41] | Subrahmanya, N.; Shin, Y. C., A variational Bayesian framework for group feature selection, Int. J. Mach. Learn. Cybern., 4, 609-619 (2013) |
| [42] | Tang, W. Y.; Mao, K. Z., Feature selection algorithm for mixed data with both nominal and continuous features, Pattern Recognit. Lett., 28, 563-571 (2007) |
| [43] | Tsang, E. C.C.; Chen, D. G.; Yeung, D. S.; Wang, X. Z.; Lee, J. W.T., Attributes reduction using fuzzy rough sets, IEEE Trans. Fuzzy Syst., 16, 1130-1141 (2008) |
| [44] | Valverde, L., On the structure of \(F\)-indistinguishability operators, Fuzzy Sets Syst., 17, 313-328 (1985) · Zbl 0609.04002 |
| [45] | Wang, G. Y.; Yu, H.; Yang, D. C., Decision table reduction based on conditional information entropy, Chin. J. Comput., 25, 759-766 (2002) |
| [46] | Wang, H., Nearest neighbors by neighborhood counting, IEEE Trans. Pattern Anal. Mach. Intell., 28, 942-953 (2006) |
| [47] | Wang, R.; Kwong, S.; Wang, X. Z.; Jiang, Q. S., Segment based decision tree induction with continuous valued attributes, IEEE Trans. Cybern., 45, 1262-1275 (2015) |
| [48] | Wang, X. Z., Learning from big data with uncertainty-editorial, J. Intell. Fuzzy Syst., 28, 2329-2330 (2015) |
| [49] | Wang, X. Z.; Ashfaq, R. A.R.; Fu, A. M., Fuzziness based sample categorization for classifier performance improvement, J. Intell. Fuzzy Syst., 29, 1185-1196 (2015) |
| [50] | Wei, P.; Ma, P. J.; Hu, Q. H.; Su, X. H.; Ma, C. Q., Comparative analysis on margin based feature selection algorithms, Int. J. Mach. Learn. Cybern., 5, 339-367 (2014) |
| [51] | Wong, A. K.C.; Chiu, D. K.Y., Synthesizing statistical knowledge from incomplete mixed-mode data, IEEE Trans. Pattern Anal. Mach. Intell., 9, 796-805 (1987) |
| [52] | Wu, W. Z.; Leung, Y.; Mi, J. S., On characterization of \((I, T)\)-fuzzy rough approximation operators, Fuzzy Sets Syst., 154, 76-102 (2005) · Zbl 1074.03027 |
| [53] | Wu, W. Z.; Leung, Y.; Shao, M. W., Generalized fuzzy rough approximation operators determined by fuzzy implicators, Int. J. Approx. Reason., 54, 1388-1409 (2013) · Zbl 1316.68198 |
| [54] | Wu, W. Z.; Mi, J. S.; Zhang, W. X., Generalized fuzzy rough sets, Inf. Sci., 151, 263-282 (2003) · Zbl 1019.03037 |
| [55] | Wu, W. Z.; Zhang, W. X., Constructive and axiomatic approaches of fuzzy approximation operators, Inf. Sci., 159, 233-254 (2004) · Zbl 1071.68095 |
| [56] | Xie, Z. X.; Xu, Y., Sparse group LASSO based uncertain feature selection, Int. J. Mach. Learn. Cybern., 5, 201-210 (2014) |
| [57] | Yao, Y. Q.; Mi, J. S.; Li, Z. J., A novel variable precision \((\theta, \sigma)\)-fuzzy rough set model based on fuzzy granules, Fuzzy Sets Syst., 236, 58-72 (2014) · Zbl 1315.68248 |
| [58] | Yeung, D. S.; Chen, D. G.; Tsang, E. C.C.; Lee, J. W.T.; Wang, X. Z., On the generalization of fuzzy rough sets, IEEE Trans. Fuzzy Syst., 13, 343-361 (2005) |
| [59] | Yu, L.; Liu, H., Efficient feature selection via analysis of relevance and redundancy, J. Mach. Learn. Res., 5, 1205-1224 (2004) · Zbl 1222.68340 |
| [60] | Zeng, A. P.; Li, T. R.; Liu, D.; Zhang, J. B.; Chen, H. M., A fuzzy rough set approach for incremental feature selection on hybrid information systems, Fuzzy Sets and Systems, 258, 39-60 (2015) · Zbl 1335.68274 |
| [61] | Zhang, H. G.; Bai, X.; Zhou, J.; Cheng, J.; Zhao, H. J., Object detection via structural feature selection and shape model, IEEE Trans. Image Process., 22, 4984-4995 (2013) · Zbl 1373.94824 |
| [62] | Zhang, X.; Mei, C. L.; Chen, D. G.; Li, J. H., Multi-confidence rule acquisition and confidence-preserved attribute reduction in interval-valued decision systems, Int. J. Approx. Reason., 55, 1787-1804 (2014) · Zbl 1433.68458 |
| [63] | Zhao, S. Y.; Tsang, E. C.C.; Chen, D. G., The model of fuzzy variable precision rough sets, IEEE Trans. Fuzzy Syst., 17, 451-467 (2009) |
| [64] | Zhu, Z. X.; Ong, Y. S.; Dash, M., Wrapper-filter feature selection algorithm using a memetic framework, IEEE Trans. Syst. Man Cybern. - Part B: Cybern., 37, 70-76 (2007) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
