Benchmarking state-of-the-art classification algorithms for credit scoring: an update of research. (English) Zbl 1346.90835
Summary: Many years have passed since Baesens et al. published their benchmarking study of classification algorithms in credit scoring [B. Baesens et al., J. Oper. Res. Soc. 54, No. 6, 627–635 (2003; Zbl 1097.91516)]. The interest in prediction methods for scorecard development is unbroken. However, there have been several advancements including novel learning methods, performance measures and techniques to reliably compare different classifiers, which the credit scoring literature does not reflect. To close these research gaps, we update the study of Baesens et al. and compare several novel classification algorithms to the state-of-the-art in credit scoring. In addition, we examine the extent to which the assessment of alternative scorecards differs across established and novel indicators of predictive accuracy. Finally, we explore whether more accurate classifiers are managerial meaningful. Our study provides valuable insight for professionals and academics in credit scoring. It helps practitioners to stay abreast of technical advancements in predictive modeling. From an academic point of view, the study provides an independent assessment of recent scoring methods and offers a new baseline to which future approaches can be compared.
MSC:
| 90C60 | Abstract computational complexity for mathematical programming problems |
| 62H30 | Classification and discrimination; cluster analysis (statistical aspects) |
| 62P05 | Applications of statistics to actuarial sciences and financial mathematics |
| 91G40 | Credit risk |
Citations:
Zbl 1097.91516Software:
UCI-mlReferences:
| [1] | Abdou, H. A., Genetic programming for credit scoring: The case of Egyptian public sector banks, Expert Systems with Applications, 36, 11402-11417 (2009) |
| [2] | Abdou, H. A.; Pointon, J.; El-Masry, A., Neural nets versus conventional techniques in credit scoring in Egyptian banking, Expert Systems with Applications, 35, 1275-1292 (2008) |
| [3] | Abellán, J.; Mantas, C. J., Improving experimental studies about ensembles of classifiers for bankruptcy prediction and credit scoring, Expert Systems with Applications, 41, 3825-3830 (2014) |
| [4] | Akkoc, S., An empirical comparison of conventional techniques, neural networks and the three stage hybrid Adaptive Neuro Fuzzy Inference System (ANFIS) model for credit scoring analysis: The case of Turkish credit card data, European Journal of Operational Research, 222, 168-178 (2012) |
| [5] | Andreeva, G., European generic scoring models using survival analysis, Journal of the Operational Research Society, 57, 1180-1187 (2006) · Zbl 1121.90072 |
| [6] | Atish, P. S.; Jerrold, H. M., Evaluating and tuning predictive data mining models using receiver operating characteristic curves, Journal of Management Information Systems, 21, 249-280 (2004) |
| [7] | Baesens, B.; Van Gestel, T.; Viaene, S.; Stepanova, M.; Suykens, J.; Vanthienen, J., Benchmarking state-of-the-art classification algorithms for credit scoring, Journal of the Operational Research Society, 54, 627-635 (2003) · Zbl 1097.91516 |
| [8] | Bellotti, T.; Crook, J., Support vector machines for credit scoring and discovery of significant features, Expert Systems with Applications, 36, 3302-3308 (2009) |
| [9] | Breiman, L., Bagging predictors, Machine Learning, 24, 123-140 (1996) · Zbl 0858.68080 |
| [10] | Breiman, L., Random forests, Machine Learning, 45, 5-32 (2001) · Zbl 1007.68152 |
| [11] | Brown, I.; Mues, C., An experimental comparison of classification algorithms for imbalanced credit scoring data sets, Expert Systems with Applications, 39, 3446-3453 (2012) |
| [12] | Calabrese, R., Downturn loss given default: Mixture distribution estimation, European Journal of Operational Research, 237, 271-277 (2014) · Zbl 1304.91247 |
| [13] | Caruana, R.; Munson, A.; Niculescu-Mizil, A., Getting the most out of ensemble selection, Proceedings of the 6th international conference on data mining, 828-833 (2006), IEEE Computer Society: IEEE Computer Society Hong Kong, China |
| [14] | Chen, W.; Ma, C.; Ma, L., Mining the customer credit using hybrid support vector machine technique, Expert Systems with Applications, 36, 7611-7616 (2009) |
| [15] | Crone, S. F.; Lessmann, S.; Stahlbock, R., The impact of preprocessing on data mining: An evaluation of classifier sensitivity in direct marketing, European Journal of Operational Research, 173, 781-800 (2006) · Zbl 1120.90349 |
| [16] | Crook, J. N.; Edelman, D. B.; Thomas, L. C., Recent developments in consumer credit risk assessment, European Journal of Operational Research, 183, 1447-1465 (2007) · Zbl 1138.91493 |
| [17] | Demšar, J., Statistical comparisons of classifiers over multiple data sets, Journal of Machine Learning Research, 7, 1-30 (2006) · Zbl 1222.68184 |
| [18] | Dietterich, T. G., Approximate statistical tests for comparing supervised classification learning, Neural Computation, 10, 1895-1923 (1998) |
| [19] | Dirick, L.; Claeskens, G.; Baesens, B., An Akaike information criterion for multiple event mixture cure models, European Journal of Operational Research, 241, 449-457 (2015) · Zbl 1341.62076 |
| [20] | Fawcett, T., An introduction to ROC analysis, Pattern Recognition Letters, 27, 861-874 (2006) |
| [21] | Finlay, S., Credit scoring for profitability objectives, European Journal of Operational Research, 202, 528-537 (2009) · Zbl 1175.90269 |
| [22] | Finlay, S., Multiple classifier architectures and their application to credit risk assessment, European Journal of Operational Research, 210, 368-378 (2011) |
| [23] | Freund, Y.; Schapire, R. E., Experiments with a new boosting algorithm, (Saitta, L., Proceedings of the 13th international conference on machine learning (1996), Morgan Kaufmann: Morgan Kaufmann Bari, Italy), 148-156 |
| [24] | Friedman, J. H., Stochastic gradient boosting, Computational Statistics & Data Analysis, 38, 367-378 (2002) · Zbl 1072.65502 |
| [25] | García, S.; Fernández, A.; Luengo, J.; Herrera, F., Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power, Information Sciences, 180, 2044-2064 (2010) |
| [26] | García, S.; Herrera, F., An extension on “Statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons, Journal of Machine Learning Research, 9, 2677-2694 (2008) · Zbl 1225.68178 |
| [27] | Gong, R.; Huang, S. H., A Kolmogorov-Smirnov statistic based segmentation approach to learning from imbalanced datasets: With application in property refinance prediction, Expert Systems with Applications, 39, 6192-6200 (2012) |
| [28] | Guang-Bin, H.; Lei, C.; Chee-Kheong, S., Universal approximation using incremental constructive feedforward networks with random hidden nodes, IEEE Transactions on Neural Networks, 17, 879-892 (2006) |
| [29] | Hall, M. A., Correlation-based feature selection for discrete and numeric class machine learning, (Langley, P., Proceedings of the 17th international conference on machine learning (2000), Morgan Kaufmann: Morgan Kaufmann Stanford, CA, USA), 359-366 |
| [30] | Hand, D. J., Good practice in retail credit scorecard assessment, Journal of the Operational Research Society, 56, 1109-1117 (2005) · Zbl 1097.91523 |
| [31] | Hand, D. J., Classifier technology and the illusion of progress, Statistical Science, 21, 1-14 (2006) · Zbl 1426.62188 |
| [32] | Hand, D. J., Measuring classifier performance: A coherent alternative to the area under the ROC curve, Machine Learning, 77, 103-123 (2009) · Zbl 1470.62085 |
| [33] | Hand, D. J.; Anagnostopoulos, C., When is the area under the receiver operating characteristic curve an appropriate measure of classifier performance?, Pattern Recognition Letters, 34, 492-495 (2013) |
| [34] | Hand, D. J.; Henley, W. E., Statistical classification models in consumer credit scoring: A review, Journal of the Royal Statistical Society: Series A (General), 160, 523-541 (1997) · Zbl 0893.90124 |
| [35] | Hand, D. J.; Sohn, S. Y.; Kim, Y., Optimal bipartite scorecards, Expert Systems with Applications, 29, 684-690 (2005) |
| [36] | He, J.; Shi, Y.; Xu, W., Classifications of credit cardholder behavior by using multiple criteria non-linear programming, (Shi, Y.; Xu, W.; Chen, Z., Chinese academy of sciences symposium on data mining and knowledge management, vol. 3327 (2004), Springer: Springer Beijing, China), 154-163 |
| [37] | Hens, A. B.; Tiwari, M. K., Computational time reduction for credit scoring: An integrated approach based on support vector machine and stratified sampling method, Expert Systems with Applications, 39, 6774-6781 (2012) |
| [38] | Hernández-Orallo, J.; Flach, P. A.; Ramirez, C. F., Brier curves: A new cost-based visualisation of classifier performance, (Getoor, L.; Scheffer, T., Proceedings of the 28th international conference on machine learning (2011), Omnipress: Omnipress Bellevue, WA, USA), 585-592 |
| [39] | Hofer, V., Adapting a classification rule to local and global shift when only unlabelled data are available, European Journal of Operational Research, 243, 177-189 (2015) · Zbl 1347.62111 |
| [40] | Hsieh, N.-C.; Hung, L.-P., A data driven ensemble classifier for credit scoring analysis, Expert Systems with Applications, 37, 534-545 (2010) |
| [41] | Huang, C.-L.; Chen, M.-C.; Wang, C.-J., Credit scoring with a data mining approach based on support vector machines, Expert Systems with Applications, 33, 847-856 (2007) |
| [42] | Huang, Y.-M.; Hunga, C.-M.; Jiau, H. C., Evaluation of neural networks and data mining methods on a credit assessment task for class imbalance problem, Nonlinear Analysis: Real World Applications, 7, 720-747 (2006) · Zbl 1160.91368 |
| [43] | Ko, A. H.R.; Sabourin, R.; Britto, J. A.S., From dynamic classifier selection to dynamic ensemble selection, Pattern Recognition, 41, 1735-1748 (2008) |
| [44] | Kruppa, J.; Schwarz, A.; Arminger, G.; Ziegler, A., Consumer credit risk: Individual probability estimates using machine learning, Expert Systems with Applications, 40, 5125-5131 (2013) |
| [45] | Kumar, P. R.; Ravi, V., Bankruptcy prediction in banks and firms via statistical and intelligent techniques—A review, European Journal of Operational Research, 180, 1-28 (2007) · Zbl 1114.91305 |
| [46] | Lee, T.-S.; Chen, I. F., A two-stage hybrid credit scoring model using artificial neural networks and multivariate adaptive regression splines, Expert Systems with Applications, 28, 743-752 (2005) |
| [47] | Lee, T.-S.; Chiu, C.-C.; Chou, Y.-C.; Lu, C.-J., Mining the customer credit using classification and regression tree and multivariate adaptive regression splines, Computational Statistics & Data Analysis, 50, 1113-1130 (2006) · Zbl 1431.62645 |
| [48] | Li, J.; Wei, L.; Li, G.; Xu, W., An evolution strategy-based multiple kernels multi-criteria programming approach: The case of credit decision making, Decision Support Systems, 51, 292-298 (2011) |
| [49] | Li, S.-T.; Shiue, W.; Huang, M.-H., The evaluation of consumer loans using support vector machines, Expert Systems with Applications, 30, 772-782 (2006) |
| [50] | Li, S.; Tsang, I. W.; Chaudhari, N. S., Relevance vector machine based infinite decision agent ensemble learning for credit risk analysis, Expert Systems with Applications, 39, 4947-4953 (2012) |
| [51] | Lichman, M., UCI machine learning repository <http://archive.ics.uci.edu/ml/> (2013), School of Information and Computer Science , University of California: School of Information and Computer Science , University of California Irvine, CA, Accessed 16.02.15 |
| [52] | Liu, F.; Hua, Z.; Lim, A., Identifying future defaulters: A hierarchical Bayesian method, European Journal of Operational Research, 241, 202-211 (2015) · Zbl 1338.91145 |
| [53] | Malhotra, R.; Malhotra, D. K., Evaluating consumer loans using neural networks, Omega, 31, 83-96 (2003) |
| [54] | Marqués, A. I.; García, V.; Sánchez, J. S., Exploring the behaviour of base classifiers in credit scoring ensembles, Expert Systems with Applications, 39, 10244-10250 (2012) |
| [55] | Marqués, A. I.; García, V.; Sánchez, J. S., Two-level classifier ensembles for credit risk assessment, Expert Systems with Applications, 39, 10916-10922 (2012) |
| [56] | Martens, D.; Van Gestel, T.; De Backer, M.; Haesen, R.; Vanthienen, J.; Baesens, B., Credit rating prediction using Ant Colony Optimization, Journal of the Operational Research Society, 61, 561-573 (2010) |
| [57] | Nanni, L.; Lumini, A., An experimental comparison of ensemble of classifiers for bankruptcy prediction and credit scoring, Expert Systems with Applications, 36, 3028-3033 (2009) |
| [58] | Ong, C.-S.; Huanga, J.-J.; Tzeng, G.-H., Building credit scoring models using genetic programming, Expert Systems with Applications, 29, 41-47 (2005) |
| [59] | Paleologo, G.; Elisseeff, A.; Antonini, G., Subagging for credit scoring models, European Journal of Operational Research, 201, 490-499 (2010) |
| [60] | Partalas, I.; Tsoumakas, G.; Vlahavas, I., Pruning an ensemble of classifiers via reinforcement learning, Neurocomputing, 72, 1900-1909 (2009) |
| [61] | Partalas, I.; Tsoumakas, G.; Vlahavas, I., An ensemble uncertainty aware measure for directed hill climbing ensemble pruning, Machine Learning, 81, 257-282 (2010) |
| [62] | Ping, Y.; Yongheng, L., Neighborhood rough set and SVM based hybrid credit scoring classifier, Expert Systems with Applications, 38, 11300-11304 (2011) |
| [63] | Platt, J. C., Probabilities for support vector machines, (Smola, A.; Bartlett, P.; Schölkopf, B.; Schuurmans, D., Advances in large margin classifiers (2000), MIT Press: MIT Press Cambridge), 61-74 · Zbl 0988.68145 |
| [64] | Pundir, S.; Seshadri, R., A novel concept of partial lorenz curve and partial gini index, International Journal of Engineering, Science and Innovative Technology, 1, 296-301 (2012) |
| [65] | Rodriguez, J. J.; Kuncheva, L. I.; Alonso, C. J., Rotation forest: A new classifier ensemble method, IEEE Transactions on Pattern Analysis and Machine Intelligence, 28, 1619-1630 (2006) |
| [66] | Sinha, A. P.; Zhao, H., Incorporating domain knowledge into data mining classifiers: An application in indirect lending, Decision Support Systems, 46, 287-299 (2008) |
| [67] | So, M. M.C.; Thomas, L. C., Modelling the profitability of credit cards by Markov decision processes, European Journal of Operational Research, 212, 123-130 (2011) |
| [68] | Sohn, S. Y.; Ju, Y. H., Updating a credit-scoring model based on new attributes without realization of actual data, European Journal of Operational Research, 234, 119-126 (2014) · Zbl 1305.62362 |
| [69] | Šušteršič, M.; Mramor, D.; Zupan, J., Consumer credit scoring models with limited data, Expert Systems with Applications, 36, 4736-4744 (2009) |
| [70] | Thomas, L. C., Consumer finance: Challenges for operational research, Journal of the Operational Research Society, 61, 41-52 (2010) · Zbl 1193.91080 |
| [71] | Thomas, L. C.; Edelman, D. B.; Crook, J. N., Credit scoring and its applications (2002), SIAM: SIAM Philadelphia · Zbl 1001.91052 |
| [72] | Tong, E. N.C.; Mues, C.; Thomas, L. C., Mixture cure models in credit scoring: if and when borrowers default, European Journal of Operational Research, 218, 132-139 (2012) · Zbl 1244.91099 |
| [73] | Tsai, C.-F., Combining cluster analysis with classifier ensembles to predict financial distress, Information Fusion, 16, 46-58 (2014) |
| [74] | Tsai, C.-F.; Wu, J.-W., Using neural network ensembles for bankruptcy prediction and credit scoring, Expert Systems with Applications, 34, 2639-2649 (2008) |
| [75] | Tsai, M.-C.; Lin, S.-P.; Cheng, C.-C.; Lin, Y.-P., The consumer loan default predicting model—An application of DEA-DA and neural network, Expert Systems with Applications, 36, 11682-11690 (2009) |
| [76] | Twala, B., Multiple classifier application to credit risk assessment, Expert Systems with Applications, 37, 3326-3336 (2010) |
| [77] | Verbeke, W.; Dejaeger, K.; Martens, D.; Hur, J.; Baesens, B., New insights into churn prediction in the telecommunication sector: A profit driven data mining approach, European Journal of Operational Research, 218, 211-229 (2012) |
| [78] | Verbraken, T.; Bravo, C.; Weber, R.; Baesens, B., Development and application of consumer credit scoring models using profit-based classification measures, European Journal of Operational Research, 238, 505-513 (2014) · Zbl 1338.91146 |
| [79] | Viaene, S.; Dedene, G., Cost-sensitive learning and decision making revisited, European Journal of Operational Research, 166, 212-220 (2004) · Zbl 1066.90537 |
| [80] | Wang, G.; Hao, J.; Ma, J.; Jiang, H., A comparative assessment of ensemble learning for credit scoring, Expert Systems with Applications, 38, 223-230 (2011) |
| [81] | West, D.; Dellana, S.; Qian, J., Neural network ensemble strategies for financial decision applications, Computers & Operations Research, 32, 2543-2559 (2005) · Zbl 1090.91547 |
| [82] | Woloszynski, T.; Kurzynski, M., A probabilistic model of classifier competence for dynamic ensemble selection, Pattern Recognition, 44, 2656-2668 (2011) · Zbl 1218.68155 |
| [83] | Xiao, W.; Zhao, Q.; Fei, Q., A comparative study of data mining methods in consumer loans credit scoring management, Journal of Systems Science and Systems Engineering, 15, 419-435 (2006) |
| [84] | Xu, X.; Zhou, C.; Wang, Z., Credit scoring algorithm based on link analysis ranking with support vector machine, Expert Systems with Applications, 36, 2625-2632 (2009) |
| [85] | Yang, Y., Adaptive credit scoring with kernel learning methods, European Journal of Operational Research, 183, 1521-1536 (2007) · Zbl 1138.91486 |
| [86] | Yao, X.; Crook, J.; Andreeva, G., Support vector regression for loss given default modelling, European Journal of Operational Research, 240, 528-538 (2015) · Zbl 1357.91050 |
| [87] | Yap, B. W.; Ong, S. H.; Husain, N. H.M., Using data mining to improve assessment of credit worthiness via credit scoring models, Expert Systems with Applications, 38, 13274-13283 (2011) |
| [88] | Yu, L.; Wang, S.; Lai, K. K., Credit risk assessment with a multistage neural network ensemble learning approach, Expert Systems with Applications, 34, 1434-1444 (2008) |
| [89] | Yu, L.; Wang, S.; Lai, K. K., An intelligent-agent-based fuzzy group decision making model for financial multicriteria decision support: The case of credit scoring, European Journal of Operational Research, 195, 942-959 (2009) · Zbl 1161.90424 |
| [90] | Yu, L.; Yao, X.; Wang, S.; Lai, K. K., Credit risk evaluation using a weighted least squares SVM classifier with design of experiment for parameter selection, Expert Systems with Applications, 38, 15392-15399 (2011) |
| [91] | Yu, L.; Yue, W.; Wang, S.; Lai, K. K., Support vector machine based multiagent ensemble learning for credit risk evaluation, Expert Systems with Applications, 37, 1351-1360 (2010) |
| [92] | Zhang, D.; Zhou, X.; Leung, S. C.H.; Zheng, J., Vertical bagging decision trees model for credit scoring, Expert Systems with Applications, 37, 7838-7843 (2010) |
| [93] | Zhang, J.; Shi, Y.; Zhang, P., Several multi-criteria programming methods for classification, Computers & Operations Research, 36, 823-836 (2009) · Zbl 1165.90638 |
| [94] | Zhou, L.; Lai, K. K.; Yu, L., Least Squares Support Vector Machines ensemble models for credit scoring, Expert Systems with Applications, 37, 127-133 (2010) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
