An integrated surrogate model constructing method: annealing combinable Gaussian process. (English) Zbl 1532.93028
Summary: Surrogate models are widely used to mimic complex systems to reduce the high experimental cost. As the system becomes high-dimensional and complex, there is an increasing demand for building relatively simplified surrogate models that capture key variables and represent complex interactions. This study proposes the annealing combinable Gaussian process, an integrated solution for identifying key variables and constructing the high-precision surrogate model. Firstly, to identify redundant variables, this study optimises variables selection with a modified simulated annealing algorithm over the complete model space. This process is called the outer loop. Secondly, to improve the model accuracy and structure, this study constructs a non-parametric Gaussian process model with additive or multiplicative kernel, effectively extracting high-order interactions. Simultaneously, a Markov chain is proposed to sample the model space. The conditional entropy is used as the scoring rule for the simulated annealing algorithm of the outer loop. It is named the inner loop. This study also discusses the rationality of conditional entropy as a criterion. The annealing combinable Gaussian process performs well in various application scenarios, including regression and classification problems. Finally, our method is implemented with the particle-in-cell simulation to find out the key physics parameters.
References:
| [1] | Aremu, O. O.; Cody, R. A.; Hyland-Wood, D.; McAree, P. R., A relative entropy based feature selection framework for asset data in predictive maintenance, Comput. Ind. Eng., 145, Article 106536 pp. (2020) |
| [2] | Calvo, B.; Santafé Rodrigo, G., scmamp: Statistical comparison of multiple algorithms in multiple problems, R J., 8 (2016) |
| [3] | Chandrashekar, G.; Sahin, F., A survey on feature selection methods, Comput. Electr. Eng., 40, 16-28 (2014) |
| [4] | Chang, C. C.; Lin, C. J., LIBSVM: A library for support vector machines, ACM Trans. Interact. Intell. Syst., 2, 1-27 (2011) |
| [5] | Dawid, A. P.; Musio, M., Bayesian model selection based on proper scoring rules, Bayesian Anal., 10, 479-499 (2015) · Zbl 1335.62017 |
| [6] | Demšar, J., Statistical comparisons of classifiers over multiple data sets, J. Mach. Learn. Res., 7, 1-30 (2006) · Zbl 1222.68184 |
| [7] | Descloux, P.; Sardy, S., Model selection with lasso-zero: Adding straw to the haystack to better find needles, J. Comput. Graph. Stat., 1-14 (2021) |
| [8] | Distante, A.; Distante, C.; Distante, Wheeler, Handbook of Image Processing and Computer Vision (2020), Springer · Zbl 1435.68002 |
| [9] | Djuric, N.; Lan, L.; Vucetic, S.; Wang, Z., Budgetedsvm: A toolbox for scalable SVM approximations, J. Mach. Learn. Res., 14, 3813-3817 (2013) · Zbl 1317.68153 |
| [10] | Efron, B.; Hastie, T.; Johnstone, I.; Tibshirani, R., Least angle regression, Ann. Stat., 32, 407-499 (2004) · Zbl 1091.62054 |
| [11] | Emary, E.; Zawbaa, H. M.; Hassanien, A. E., Binary grey wolf optimization approaches for feature selection, Neurocomputing, 172, 371-381 (2016) |
| [12] | Finner, H., On a monotonicity problem in step-down multiple test procedures, J. Am. Stat. Assoc., 88, 920-923 (1993) · Zbl 0799.62077 |
| [13] | García, S.; Fernández, A.; Luengo, J.; Herrera, F., Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power, Inf. Sci., 180, 2044-2064 (2010) |
| [14] | S. Garcia, F. Herrera, An extension on “statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons, J. Mach. Learn. Res. 9 (2008). · Zbl 1225.68178 |
| [15] | Gardner, J.; Guo, C.; Weinberger, K.; Garnett, R.; Grosse, R., Discovering and exploiting additive structure for Bayesian optimization, Artif. Intell. Stat., 1311-1319 (2017) |
| [16] | Van de Geer, S.; Bühlmann, P.; Zhou, S., The adaptive and the thresholded lasso for potentially misspecified models (and a lower bound for the lasso), Electron. J. Stat., 5, 688-749 (2011) · Zbl 1274.62471 |
| [17] | Goes, J.; Zhang, T.; Arora, R.; Lerman, G., Robust stochastic principal component analysis, Artificial Intelligence and Statistics, PMLR, 266-274 (2014) |
| [18] | Guan, D.; Yuan, W.; Lee, Y. K.; Najeebullah, K.; Rasel, M. K., A review of ensemble learning based feature selection, IETE Tech. Rev., 31, 190-198 (2014) |
| [19] | Guyon, I.; Elisseeff, A., An introduction to variable and feature selection, J. Mach. Learn. Res., 3, 1157-1182 (2003) · Zbl 1102.68556 |
| [20] | Hastie, T.; Tibshirani, R.; Wainwright, M., Statistical learning with sparsity: the lasso and generalizations (2019), Chapman and Hall/CRC |
| [21] | Hsu, H. H.; Hsieh, C. W.; Lu, M. D., Hybrid feature selection by combining filters and wrappers, Expert Syst. Appl., 38, 8144-8150 (2011) |
| [22] | Javanmard, A.; Montanari, A., Debiasing the lasso: Optimal sample size for Gaussian designs, Ann. Stat., 46, 2593-2622 (2018) · Zbl 1407.62270 |
| [23] | Kirk, J. G.; Bell, A.; Arka, I., Pair production in counter-propagating laser beams, Plasma Phys. Controlled Fusionn, 51, Article 085008 pp. (2009) |
| [24] | Li, J.; Cheng, K.; Wang, S.; Morstatter, F.; Trevino, R. P.; Tang, J.; Liu, H., Feature selection: A data perspective, ACM Comput. Surv., 50, 1-45 (2017) |
| [25] | Mac Nally, R.; Duncan, R. P.; Thomson, J. R.; Yen, J. D., Model selection using information criteria, but is the “best” model any good?, J. Appl. Ecol., 55, 1441-1444 (2018) |
| [26] | Maldonado, S.; López, J., Dealing with high-dimensional class-imbalanced datasets: Embedded feature selection for svm classification, Appl. Soft Comput., 67, 94-105 (2018) |
| [27] | Pesaresi, M.; Syrris, V.; Julea, A., Benchmarking of the symbolic machine learning classifier with state of the art image classification methods-Application to remote sensing imagery, Technical Report (2015) |
| [28] | Raftery, A. E.; Madigan, D.; Hoeting, J. A., Bayesian model averaging for linear regression models, J. Am. Stat. Assoc., 92, 179-191 (1997) · Zbl 0888.62026 |
| [29] | Ridgers, C.; Brady, C.; Duclous, R.; Kirk, J.; Bennett, K.; Arber, T.; Robinson, A.; Bell, A., Dense electron-positron plasmas and ultraintense γ)rays from laser-irradiated solids, Phys. Rev. Lett., 108, Article 165006 pp. (2012) |
| [30] | Roy, S. S.; Samanta, K.; Modak, S.; Chatterjee, S.; Bose, R., Cross spectrum aided deep feature extraction based neuromuscular disease detection framework, IEEE Sens. Lett., 4, 1-4 (2020) |
| [31] | Saleh, A. M.E.; Arashi, M.; Kibria, B. G., Theory of ridge regression estimation with applications, vol. 285 (2019), John Wiley & Sons · Zbl 1409.62005 |
| [32] | B. Sang, H. Chen, L. Yang, T. Li, W. Xu, Incremental feature selection using a conditional entropy based on fuzzy dominance neighborhood rough sets, IEEE Trans. Fuzzy. Syst. (2021). |
| [33] | Sharma, P. N.; Shmueli, G.; Sarstedt, M.; Danks, N.; Ray, S., Prediction-oriented model selection in partial least squares path modeling, Decis. Sci. (2019) |
| [34] | Sigillito, V. G.; Wing, S. P.; Hutton, L. V.; Baker, K. B., Classification of radar returns from the ionosphere using neural networks, Johns Hopkins APL Tech. Dig., 10, 262-266 (1989) |
| [35] | Somol, P.; Pudil, P.; Kittler, J., Fast branch & bound algorithms for optimal feature selection, IEEE Trans. Pattern Anal. Mach. Intell., 26, 900-912 (2004) |
| [36] | Sorostinean, R.; Gellert, A.; Pirvu, B. C., Assembly assistance system with decision trees and ensemble learning, Sensors, 21, 3580 (2021) |
| [37] | Su, W.; Bogdan, M.; Candes, E., False discoveries occur early on the lasso path, Ann. Stat., 2133-2150 (2017) · Zbl 1459.62142 |
| [38] | Sun, T.; Zhang, C. H., Scaled sparse linear regression, Biometrika, 99, 879-898 (2012) · Zbl 1452.62515 |
| [39] | Tibshirani, R., Regression shrinkage and selection via the lasso, J.R. Stat. Soc. B, 58, 267-288 (1996) · Zbl 0850.62538 |
| [40] | Vergara, J. R.; Estévez, P. A., A review of feature selection methods based on mutual information, Neural. Comput. Appl., 24, 175-186 (2014) |
| [41] | Vrieze, S. I., Model selection and psychological theory: a discussion of the differences between the Akaike information criterion (aic) and the Bayesian information criterion (bic), Psychol. methods, 17, 228 (2012) |
| [42] | Wang, B.; Duan, X.; Yan, L.; Deng, J.; Chen, J., Rapidly tuning the PID controller based on the regional surrogate model technique in the UAV formation, Entropy, 22, 527 (2020) |
| [43] | Webb, G. I.; Boughton, J. R.; Wang, Z., Not so naive Bayes: aggregating one-dependence estimators, Mach. Learn., 58, 5-24 (2005) · Zbl 1075.68078 |
| [44] | Xue, Y.; Xue, B.; Zhang, M., Self-adaptive particle swarm optimization for large-scale feature selection in classification, ACM Trans. Knowl. Discovery Data, 13, 1-27 (2019) |
| [45] | Yao, C.; Liu, Y. F.; Jiang, B.; Han, J.; Han, J., LLE score: A new filter-based unsupervised feature selection method based on nonlinear manifold embedding and its application to image recognition, IEEE Trans. Image Process., 26, 5257-5269 (2017) · Zbl 1409.94710 |
| [46] | Yuan, M.; Lin, Y., Model selection and estimation in regression with grouped variables, J. R. Stat. Soc. B, 68, 49-67 (2006) · Zbl 1141.62030 |
| [47] | Yun, Y. H.; Bin, J.; Liu, D. L.; Xu, L.; Yan, T. L.; Cao, D. S.; Xu, Q. S., A hybrid variable selection strategy based on continuous shrinkage of variable space in multivariate calibration, Anal. Chim. Acta, 1058, 58-69 (2019) |
| [48] | Zorarpaci, E.; Özel, S. A., A hybrid approach of differential evolution and artificial bee colony for feature selection, Expert Syst. Appl., 62, 91-103 (2016) |
| [49] | Zou, H.; Hastie, T., Regularization and variable selection via the elastic net, J. R. Stat. Soc. B, 67, 301-320 (2005) · Zbl 1069.62054 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
