A new principle for tuning-free Huber regression. (English) Zbl 1524.62136
Summary: The robustification parameter, which balances bias and robustness, plays a critical role in the construction of subGaussian estimators for heavy-tailed and/or skewed data. Although the parameter can be tuned using cross-validation, in large-scale statistical problems such as high-dimensional covariance matrix estimation and large-scale multiple testing, the number of robustification parameters increases with the dimensionality causing cross-validation to become computationally prohibitive. We propose a new data-driven principle for choosing the robustification parameter for Huber-type subGaussian estimators in three fundamental problems: mean estimation, linear regression, and sparse regression in high dimensions. Our proposal is guided by a nonasymptotic deviation analysis, and is conceptually different from cross-validation, which relies on the mean squared error to assess the fit. Extensive numerical experiments and a real-data analysis further illustrate the efficacy of the proposed methods.
MSC:
| 62F35 | Robustness and adaptive procedures (parametric inference) |
| 62H12 | Estimation in multivariate analysis |
| 62J05 | Linear regression; mixed models |
Software:
robustbaseReferences:
| [1] | Alquier, P., Cottet, V. and Lecué, G. (2019). Estimation bounds and sharp oracle inequalities of regularized procedures with Lipschitz loss functions. The Annals of Statistics 47, 2117-2144. · Zbl 1466.62289 |
| [2] | Audibert, J.-Y. and Catoni, O. (2011). Robust linear least squares regression. The Annals of Statistics 39, 2766-2794. · Zbl 1231.62126 |
| [3] | Avella-Medina, M., Battey, H. S., Fan, J. and Li, Q. (2018). Robust estimation of high-dimensional covariance and precision matrices. Biometrika 105, 271-284. · Zbl 07072412 |
| [4] | Avella-Medina, M. and Ronchetti, E. (2015). Robust statistics: A selective overview and new directions. WIREs Computational Statistics 7, 372-393. · Zbl 07912783 |
| [5] | Bickel, P. J. (1975). One-step Huber estimates in the linear model. Journal of the American Statistical Association 70, 428-434. · Zbl 0322.62038 |
| [6] | Brownlees, C., Joly, E. and Lugosi, G. (2015). Empirical risk minimization for heavy-tailed losses. The Annals of Statistics 43, 2507-2536. · Zbl 1326.62066 |
| [7] | Bühlmann, P. and van de Geer, S. (2011). Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer, Heidelberg. · Zbl 1273.62015 |
| [8] | Catoni, O. (2012). Challenging the empirical mean and empirical variance: A deviation study. Annales de l’Institut Henri Poincaré B: Probability and Statistics 48, 1148-1185. · Zbl 1282.62070 |
| [9] | Cont, R. (2001). Empirical properties of asset returns: Stylized facts and statistical issues. Quantitative Finance 1, 223-236. · Zbl 1408.62174 |
| [10] | Donoho, D. L. and Huber, P. J. (1983). The notion of breakdown point. In A Festschrift for Erich L. Lehmann (Edited by P. J. Bickel, K. Doksum and J. L. Hodges, Jr.), 157-184. Wadsworth, Belmont, CA. · Zbl 0523.62032 |
| [11] | Devroye, L., Lerasle, M., Lugosi, G. and Oliveira, R. I. (2016). Sub-Gaussian mean estimators. The Annals of Statistics 44, 2695-2725. · Zbl 1360.62115 |
| [12] | Elsener, A. and van de Geer, S. (2018). Robust low-rank matrix estimation. The Annals of Statistics 46, 3481-3509. · Zbl 1412.62068 |
| [13] | Fan, J., Li, Q. and Wang, Y. (2017). Estimation of high dimensional mean regression in the absence of symmetry and light tail assumptions. Journal of the Royal Statistical Society, Series B 79, 247-265. · Zbl 1414.62178 |
| [14] | Hampel, F. R. (1971). A general qualitative definition of robustness. The Annals of Mathematical Statistics 42, 1887-1896. · Zbl 0229.62041 |
| [15] | Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J. and Stahel, W. A. (1986). Robust Statistics: The Approach Based on Influence Functions. Wiley, New York. · Zbl 0593.62027 |
| [16] | Hastie, T., Tibshirani, R. and Wainwright, M. (2015). Statistical Learning with Sparsity: The Lasso and Generalizations. CRC Press, Boca Raton. · Zbl 1319.68003 |
| [17] | Hsu, D. and Sabato, S. (2016). Loss minimization and parameter estimation with heavy tails. Journal of Machine Learning Research 17, 1-40. · Zbl 1360.62380 |
| [18] | Huber, P. J. (1964). Robust estimation of a location parameter. Annals of Mathematical Statis-tics 35, 73-101. · Zbl 0136.39805 |
| [19] | Huber, P. J. (1973). Robust regression: Asymptotics, conjectures and Monte Carlo. The Annals of Statistics 1, 799-821. · Zbl 0289.62033 |
| [20] | Huber, P. J. (1981). Robust Statistics. Wiley, New York. · Zbl 0536.62025 |
| [21] | Huber, P. J. and Ronchetti, E. M. (2009). Robust Statistics. 2nd Edition. Wiley, New York. · Zbl 1276.62022 |
| [22] | Lambert-Lacroix, S. and Zwald, L. (2011). Robust regression through the Huber’s criterion and adaptive Lasso penalty. Electronic Journal of Statistics 5, 1015-1053. · Zbl 1274.62467 |
| [23] | Lecué, G. and Lerasle, M. (2017). Robust machine learning by median-of-means: Theory and practice. arXiv preprint arXiv:1711.10306. |
| [24] | Loh, P. (2017). Statistical consistency and asymptotic normality for high-dimensional robust M -estimators. The Annals of Statistics 45, 866-896. · Zbl 1371.62023 |
| [25] | Loh, P.-L. and Wainwright, M. J. (2015). Regularized M -estimators with nonconvexity: Statis-tical and algorithmic theory for local optima. Journal of Machine Learning Research 16, 559-616. · Zbl 1360.62276 |
| [26] | Lugosi, G. and Mendelson, S. (2016). Risk minimization by median-of-means tournaments. arXiv preprint arXiv:1608.00757. · Zbl 1436.62312 |
| [27] | Lugosi, G. and Mendelson, S. (2019). Sub-Gaussian estimators of the mean of a random vector. The Annals of Statistics 47, 783-794. · Zbl 1417.62192 |
| [28] | Mei, S., Bai, Y. and Montanari, A. (2018). The landscape of empirical risk for nonconvex losses. The Annals of Statistics 46, 2747-2774. · Zbl 1409.62117 |
| [29] | Minsker, S. (2015). Geometric median and robust estimation in Banach spaces. Bernoulli 21, 2308-2335. · Zbl 1348.60041 |
| [30] | Minsker, S. (2018). Sub-Gaussian estimators of the mean of a random matrix with heavy-tailed entries. The Annals of Statistics 46, 2871-2903. · Zbl 1418.62235 |
| [31] | Maronna, R. A., Marton, R. D., Yohai, V. J. and Salibián-Barrera, M. (2018). Robust Statistics: Theory and Methods (with R). Wiley, New York. |
| [32] | Owen, A. B. (2007). A robust hybrid of Lasso and ridge regression. Contemporary Mathematics 443, 59-72. · Zbl 1134.62047 |
| [33] | Pan, X., Sun, Q. and Zhou, W.-X. (2019). Nonconvex regularized robust regression with oracle properties in polynomial time. arXiv preprint arXiv:1907.04027. |
| [34] | Portnoy, S. and He, X. (2000). A robust journey in the new millennium. Journal of the American Statistical Association 95, 1331-1335. · Zbl 1072.62505 |
| [35] | Purdom, E. and Holmes, S. P. (2005). Error distribution for gene expression data. Statistical Applications in Genetics and Molecular Biology 4, 16. · Zbl 1083.62114 |
| [36] | She, Y. and Owen, A. B. (2011). Outlier detection using nonconvex penalized regression. Journal of the American Statistical Association 106, 626-639. · Zbl 1232.62068 |
| [37] | Sun, Q., Zhou, W.-X. and Fan, J. (2020). Adaptive Huber regression. Journal of the American Statistical Association 115, 254-265. · Zbl 1437.62250 |
| [38] | Theodossiou, P. (1998). Financial data and the skewed generalized t distribution. Management Science 44, 1650-1661. · Zbl 1001.91051 |
| [39] | Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society, Series B 58, 267-288. · Zbl 0850.62538 |
| [40] | Vershynin, R. (2012). Introduction to the non-asymptotic analysis of random matrices. In Com-pressed Sensing, 210-268. Cambridge Univ. Press, Cambridge. |
| [41] | Yi, C. and Huang, J. (2017). Semismooth Newton coordinate descent algorithm for elastic-net penalized Huber loss regression and quantile regression. Journal of Computational and Graphical Statistics 26, 547-557. |
| [42] | Zhou, W.-X., Bose, K., Fan, J. and Liu, H. (2018). A new perspective on robust M -estimation: Finite sample theory and applications to dependence-adjusted multiple testing. The Annals of Statistics 46, 1904-1931. · Zbl 1409.62154 |
| [43] | Chao Zheng |
| [44] | Wen Zhou Department of Statistics, Colorado State University, Fort Collins, Colorado 80523, USA. E-mail: riczw@stat.colostate.edu |
| [45] | Wen-Xin Zhou Department of Mathematics, University of California, San Diego, La Jolla, CA 92093, USA. E-mail: wez243@ucsd.edu (Received February 2019; accepted May 2020) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
