Approximate inference of the bandwidth in multivariate kernel density estimation. (English) Zbl 1271.62083
Summary: Kernel density estimation is a popular and widely used non-parametric method for data-driven density estimation. Its appeal lies in its simplicity and ease of implementation, as well as its strong asymptotic results regarding its convergence to the true data distribution. However, a major difficulty is the setting of the bandwidth, particularly in high dimensions and with limited amount of data. An approximate Bayesian method is proposed, based on the expectation-propagation algorithm with a likelihood obtained from a leave-one-out cross validation approach. The proposed method yields an iterative procedure to approximate the posterior distribution of the inverse bandwidth. The approximate posterior can be used to estimate the model evidence for selecting the structure of the bandwidth and approach online learning. Extensive experimental validation shows that the proposed method is competitive in terms of performance with state-of-the-art plug-in methods.
MSC:
| 62G07 | Density estimation |
Keywords:
kernel density estimation; Bayesian inference; expectation propagation; multivariate analysisReferences:
| [1] | Bishop, C. M., (Pattern Recognition and Machine Learning. Pattern Recognition and Machine Learning, Information Science and Statistics (2007), Springer) · Zbl 1107.68072 |
| [2] | Bowman, A. W., An alternative method of cross-validation for the smoothing of density estimates, Biometrika, 71, 2, 353-360 (1984) |
| [3] | Brewer, M. J., A Bayesian model for local smoothing in kernel density estimation, Statistics and Computing, 10, 4, 299-309 (2000) |
| [4] | Calderhead, B.; Girolami, M., Estimating Bayes factors via thermodynamic integration and population MCMC, Computational Statistics & Data Analysis, 53, 12, 4028-4045 (2009) · Zbl 1453.62055 |
| [5] | Cao, R.; Cuevas, A.; Manteiga, W. G., A comparative study of several smoothing methods in density estimation, Computational Statistics & Data Analysis, 17, 2, 153-176 (1994) · Zbl 0937.62518 |
| [6] | Chiu, S. T., Bandwidth selection for kernel density estimation, The Annals of Statistics, 19, 4, 1883-1905 (1991) · Zbl 0749.62022 |
| [7] | de Lima, M. S.; Atuncar, G. S., A Bayesian method to estimate the optimal bandwidth for multivariate kernel estimator, Journal of Nonparametric Statistics, 137-148 (2011) · Zbl 1359.62113 |
| [8] | Duong, T.; Cowling, A.; Koch, I.; Wand, M. P., Feature significance for multivariate kernel density estimation, Computational Statistics & Data Analysis, 52, 9, 4225-4242 (2008) · Zbl 1452.62265 |
| [9] | Duong, T.; Hazelton, M. L., Cross-validation bandwidth matrices for multivariate kernel density estimation, Scandinavian Journal of Statistics Theory and Applications, 32, 3, 485-506 (2005) · Zbl 1089.62035 |
| [10] | Ferguson, T. S., A Bayesian analysis of some nonparametric problems, The Annals of Statistics, 1, 2, 209-230 (1973) · Zbl 0255.62037 |
| [11] | Filippone, M.; Masulli, F.; Rovetta, S., Applying the possibilistic \(c\)-means algorithm in kernel-induced spaces, IEEE Transactions on Fuzzy Systems, 18, 3, 572-584 (2010) |
| [12] | Friel, N.; Pettitt, A. N., Marginal likelihood estimation via power posteriors, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70, 3, 589-607 (2008) · Zbl 05563360 |
| [13] | Gangopadhyay, A.; Cheung, K., Bayesian approach to the choice of smoothing parameter in kernel density estimation, Journal of Nonparametric Statistics, 14, 6, 655-664 (2002) · Zbl 1013.62038 |
| [14] | Gelman, A.; Rubin, D. B., Inference from iterative simulation using multiple sequences, Statistical Science, 7, 4, 457-472 (1992) · Zbl 1386.65060 |
| [15] | Hall, P., Cross-validation in density estimation, Biometrika, 69, 2, 383-390 (1982) · Zbl 0496.62038 |
| [16] | Hastings, W. K., Monte Carlo sampling methods using Markov chains and their applications, Biometrika, 57, 1, 97-109 (1970) · Zbl 0219.65008 |
| [17] | Hazelton, M. L.; Turlach, B. A., Reweighted kernel density estimation, Computational Statistics & Data Analysis, 51, 6, 3057-3069 (2007) · Zbl 1161.62352 |
| [18] | Jones, M. C., On correcting for variance inflation in kernel density estimation, Computational Statistics & Data Analysis, 11, 1, 3-15 (1991) · Zbl 0850.62344 |
| [19] | Jones, M. C.; Henderson, D. A., Maximum likelihood kernel density estimation: on the potential of convolution sieves, Computational Statistics & Data Analysis, 53, 10, 3726-3733 (2009) · Zbl 1453.62122 |
| [20] | Jones, M. C.; Marron, J. S.; Park, B. U., A simple root \(n\) bandwidth selector, The Annals of Statistics, 19, 4, 1919-1932 (1991) · Zbl 0745.62033 |
| [21] | Jones, M. C.; Marron, J. S.; Sheather, S. J., A brief survey of bandwidth selection for density estimation, Journal of the American Statistical Association, 91, 433, 401-407 (1996) · Zbl 0873.62040 |
| [22] | Kass, R. E.; Carlin, B. P.; Gelman, A.; Neal, R. M., Markov chain Monte Carlo in practice: a roundtable discussion, The American Statistician, 52, 2, 93-100 (1998) |
| [23] | Kass, R. E.; Raftery, A. E., Bayes factors, Journal of the American Statistical Association, 90, 430, 773-795 (1995) · Zbl 0846.62028 |
| [24] | Kulasekera, K. B.; Padgett, W. J., Bayes bandwidth selection in kernel density estimation with censored data, Journal of Nonparametric Statistics, 18, 2, 129-143 (2006) · Zbl 1099.62037 |
| [25] | Loader, C. R., Bandwidth selection: classical or plug-in?, The Annals of Statistics, 27, 2, 415-438 (1999) · Zbl 0938.62035 |
| [26] | Marron, J. S.; Wand, M. P., Exact mean integrated squared error, The Annals of Statistics, 20, 2, 712-736 (1992) · Zbl 0746.62040 |
| [27] | Metropolis, N.; Rosenbluth, A. W.; Rosenbluth, M. N.; Teller, A. H.; Teller, E., Equation of state calculations by fast computing machines, The Journal of Chemical Physics, 21, 6, 1087-1092 (1953) · Zbl 1431.65006 |
| [28] | Minka, T. P., Expectation propagation for approximate Bayesian inference, (Uncertainty in Artificial Intelligence: Proceedings of the Seventeenth Conference, 2001. Uncertainty in Artificial Intelligence: Proceedings of the Seventeenth Conference, 2001, August 2-5, 2001, University of Washington, Seattle, Washington (2001), Morgan Kaufmann), 362 |
| [29] | Minka, T.P., Ghahramani, Z., 2003. Expectation propagation for infinite mixtures. Tech. Rep. NIPS’03 Workshop on Nonparametric Bayesian Methods and Infinite Models, 13, Whistler, BC, Canada.; Minka, T.P., Ghahramani, Z., 2003. Expectation propagation for infinite mixtures. Tech. Rep. NIPS’03 Workshop on Nonparametric Bayesian Methods and Infinite Models, 13, Whistler, BC, Canada. |
| [30] | Neal, R.M., 1993. Probabilistic inference using Markov chain Monte Carlo methods. Tech. Rep. CRG-TR-93-1. Dept. of Computer Science, University of Toronto.; Neal, R.M., 1993. Probabilistic inference using Markov chain Monte Carlo methods. Tech. Rep. CRG-TR-93-1. Dept. of Computer Science, University of Toronto. |
| [31] | Opper, M.; Winther, O., Gaussian processes for classification: mean-field algorithms, Neural Computation, 12, 11, 2655-2684 (2000) |
| [32] | R Development Core Team, 2006. R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.; R Development Core Team, 2006. R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. |
| [33] | Rinaldo, A.; Wasserman, L., Generalized density clustering, Annals of Statistics, 38, 5, 2678-2722 (2010) · Zbl 1200.62066 |
| [34] | Sheather, S. J.; Jones, M. C., A reliable data-based bandwidth selection method for kernel density estimation, Journal of the Royal Statistical Society. Series B (Methodological), 53, 3, 683-690 (1991) · Zbl 0800.62219 |
| [35] | Silverman, B. W., (Density Estimation for Statistics and Data Analysis. Density Estimation for Statistics and Data Analysis, Chapman & Hall/CRC Monographs on Statistics & Applied Probability (1986), Chapman and Hall, CRC) · Zbl 0617.62042 |
| [36] | Skilling, J., Nested sampling for general Bayesian computation, Bayesian Analysis, 1, 4, 833-860 (2006) · Zbl 1332.62374 |
| [37] | Terrell, G. R., The maximal smoothing principle in density estimation, Journal of the American Statistical Association, 85, 410, 470-477 (1990) |
| [38] | Tran, T. N.; Wehrens, R.; Buydens, L. M.C., KNN-kernel density-based clustering for high-dimensional multivariate data, Computational Statistics & Data Analysis, 51, 2, 513-525 (2006) · Zbl 1157.62448 |
| [39] | van der Laan, M. J.; Dudoit, S.; Keles, S., Asymptotic optimality of likelihood-based cross-validation, Statistical Applications in Genetics and Molecular Biology, 3, 1 (2004) · Zbl 1038.62040 |
| [40] | Wand, M. P.; Jones, M. C., Multivariate plug-in bandwidth selection, Computational Statistics, 9, 97-116 (1994) · Zbl 0937.62055 |
| [41] | Zhang, X.; Brooks, R. D.; King, M. L., A Bayesian approach to bandwidth selection for multivariate kernel regression with an application to state-price density estimation, Journal of Econometrics, 153, 1, 21-32 (2009) · Zbl 1431.62176 |
| [42] | Zhang, X.; King, M. L.; Hyndman, R. J., A Bayesian approach to bandwidth selection for multivariate kernel density estimation, Computational Statistics and Data Analysis, 50, 11, 3009-3031 (2006) · Zbl 1445.62077 |
| [43] | Żychaluk, K.; Patil, P., A cross-validation method for data with ties in kernel density estimation, Annals of the Institute of Statistical Mathematics, 60, 1, 21-44 (2008) · Zbl 1184.62058 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
