×

Analysis of global and local optima of regularized quantile regression in high dimensions: a subgradient approach. (English) Zbl 07852007

Summary: Regularized quantile regression (QR) is a useful technique for analyzing heterogeneous data under potentially heavy-tailed error contamination in high dimensions. This paper provides a new analysis of the estimation/prediction error bounds of the global solution of \(L_1\)-regularized QR (QR-LASSO) and the local solutions of nonconvex regularized QR (QR-NCP) when the number of covariates is greater than the sample size. Our results build upon and significantly generalize the earlier work in the literature. For certain heavy-tailed error distributions and a general class of design matrices, the least-squares-based LASSO cannot achieve the near-oracle rate derived under the normality assumption no matter the choice of the tuning parameter. In contrast, we establish that QR-LASSO achieves the near-oracle estimation error rate for a broad class of models under conditions weaker than those in the literature. For QR-NCP, we establish the novel results that all local optima within a feasible region have desirable estimation accuracy. Our analysis applies to not just the hard sparsity setting commonly used in the literature, but also to the soft sparsity setting which permits many small coefficients. Our approach relies on a unified characterization of the global/local solutions of regularized QR via subgradients using a generalized Karush-Kuhn-Tucker condition. The theory of the paper establishes a key property of the subdifferential of the quantile loss function in high dimensions, which is of independent interest for analyzing other high-dimensional nonsmooth problems.

MSC:

62P20 Applications of statistics to economics

Software:

quantilogram
Full Text: DOI

References:

[1] Abadie, A., Angrist, J., & Imbens, G. (2002) Instrumental variables estimates of the effect of subsidized training on the quantiles of trainee earnings. Econometrica70(1), 91-117. · Zbl 1104.62331
[2] Angrist, J., Chernozhukov, V., & Fernández-Val, I. (2006) Quantile regression under misspecification, with an application to the US wage structure. Econometrica74(2), 539-563. · Zbl 1145.62399
[3] Arellano, M. & Bonhomme, S. (2017) Quantile selection models with an application to understanding changes in wage inequality. Econometrica85(1), 1-28. · Zbl 1420.91177
[4] Belloni, A. & Chernozhukov, V. (2011) L1-penalized quantile regression in high-dimensional sparse models. Annals of Statistics39, 82-130. · Zbl 1209.62064
[5] Belloni, A., Chernozhukov, V., & Kato, K. (2014) Uniform post-selection inference for least absolute deviation regression and other z-estimation problems. Biometrika102(1), 77-94. · Zbl 1345.62049
[6] Belloni, A., Chernozhukov, V., & Kato, K. (2019) Valid post-selection inference in high-dimensional approximately sparse quantile regression models. Journal of the American Statistical Association114(526), 749-758. · Zbl 1420.62169
[7] Bickel, P.J., Ritov, Y., & Tsybakov, A.B. (2009) Simultaneous analysis of Lasso and Dantzig selector. Annals of Statistics37(4), 1705-1732. · Zbl 1173.62022
[8] Bradic, J., Fan, J., & Wang, W. (2011) Penalized composite quasi-likelihood for ultrahigh dimensional variable selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology)73(3), 325-349. · Zbl 1411.62181
[9] Bradic, J. & Kolar, M. (2017). Uniform inference for high-dimensional quantile regression: Linear functionals and regression rank scores. Preprint, arXiv:1702.06209.
[10] Buchinsky, M. (1994) Changes in the US wage structure 1963-1987: Application of quantile regression. Econometrica62, 405-458. · Zbl 0800.90235
[11] Buchinsky, M. (1998) The dynamics of changes in the female wage distribution in the USA: A quantile regression approach. Journal of Applied Econometrics13(1), 1-30.
[12] Bunea, F., Tsybakov, A., & Wegkamp, M. (2007) Sparsity oracle inequalities for the Lasso. Electronic Journal of Statistics1, 169-194. · Zbl 1146.62028
[13] Chamberlain, G. (1994) Quantile regression, censoring, and the structure of wages. In C.A. Sims (ed.), Advances in Econometrics: Sixth World Congress, vol. 2. Cambridge University Press, pp. 171-209.
[14] Chen, X., Li, D., Li, Q., & Li, Z. (2019a) Nonparametric estimation of conditional quantile functions in the presence of irrelevant covariates. Journal of Econometrics212(2), 433-450. · Zbl 1452.62277
[15] Chen, X., Liu, W., & Zhang, Y. (2019b) Quantile regression under memory constraint. Annals of Statistics47(6), 3244-3273. · Zbl 1436.62134
[16] Chernozhukov, V. & Fernández-Val, I. (2011) Inference for extremal conditional quantile models, with an application to market and birthweight risks. The Review of Economic Studies78(2), 559-589. · Zbl 1216.62077
[17] Chernozhukov, V., Fernández-Val, I., Hahn, J., & Newey, W. (2013) Average and quantile effects in nonseparable panel models. Econometrica81(2), 535-580. · Zbl 1274.62580
[18] Donoho, D.L. & Johnstone, I.M. (1994). Minimax risk over \({l}_p\) -balls for \({l}_q\) -error. Probability Theory and Related Fields99(2), 277-303. · Zbl 0802.62006
[19] Elsener, A. & Van De Geer, S. (2018) Sharp oracle inequalities for stationary points of nonconvex penalized m-estimators. IEEE Transactions on Information Theory65(3), 1452-1472. · Zbl 1432.62062
[20] Fan, J., Fan, Y., & Barut, E. (2014) Adaptive robust variable selection. Annals of Statistics42(1), 324-351. · Zbl 1296.62144
[21] Fan, J., Li, Q., & Wang, Y. (2017) Estimation of high dimensional mean regression in the absence of symmetry and light tail assumptions. Journal of the Royal Statistical Society: Series B (Statistical Methodology)79(1), 247-265. · Zbl 1414.62178
[22] Fan, J. & Li, R. (2001) Variable selection via nonconcave penalized likelihood and its oracle property. Journal of the American Statistical Association96, 1348-1360. · Zbl 1073.62547
[23] Fan, Z. & Lian, H. (2018) Quantile regression for additive coefficient models in high dimensions. Journal of Multivariate Analysis164, 54-64. · Zbl 1499.62131
[24] Firpo, S., Fortin, N.M., & Lemieux, T. (2009) Unconditional quantile regressions. Econometrica77(3), 953-973. · Zbl 1176.62034
[25] Fitzenberger, B., Koenker, R., & Machado, J.A. (2013) Economic Applications of Quantile Regression. Springer Science & Business Media.
[26] Galvao, A.F., Lamarche, C., & Lima, L.R. (2013) Estimation of censored quantile regression for panel data with fixed effects. Journal of the American Statistical Association108(503), 1075-1089. · Zbl 06224988
[27] Graham, B.S., Hahn, J., Poirier, A., & Powell, J.L. (2018) A quantile correlated random coefficients panel data model. Journal of Econometrics206(2), 305-335. · Zbl 1452.62912
[28] Greenshtein, E., Ritov, Y. (2004) Persistence in high-dimensional linear predictor selection and the virtue of overparametrization. Bernoulli10(6), 971-988. · Zbl 1055.62078
[29] Harding, M. & Lamarche, C. (2018) A panel quantile approach to attrition bias in big data: Evidence from a randomized experiment. Journal of Econometrics211, 61-82. · Zbl 1452.62917
[30] Honda, T., Ing, C.-K., & Wu, W.-Y. (2019) Adaptively weighted group lasso for semiparametric quantile regression models. Bernoulli25(4B), 3311-3338. · Zbl 1429.62140
[31] Horowitz, J.L. & Lee, S. (2005) Nonparametric estimation of an additive quantile regression model. Journal of the American Statistical Association100(472), 1238-1249. · Zbl 1117.62355
[32] Horowitz, J.L. & Spokoiny, V.G. (2002) An adaptive, rate-optimal test of linearity for median regression models. Journal of the American Statistical Association97(459), 822-835. · Zbl 1048.62050
[33] Kai, B., Li, R., & Zou, H. (2011) New efficient estimation and variable selection methods for semiparametric varying-coefficient partially linear models. Annals of Statistics39, 305-332. · Zbl 1209.62074
[34] Kato, K. (2011) Group Lasso for high dimensional sparse quantile regression models. Preprint, arXiv:1103.1458.
[35] Koenker, R. (2017) Quantile regression: 40 years on. Annual Review of Economics9, 155-176.
[36] Koenker, R. & Bassett, G. (1978) Regression quantiles. Econometrica46, 33-50. · Zbl 0373.62038
[37] Koenker, R., Chernozhukov, V., He, X., & Peng, L. (eds.) (2017) Handbook of Quantile Regression. Chapman and Hall/CRC.
[38] Koenker, R. & Xiao, Z. (2006) Quantile autoregression. Journal of the American Statistical Association101(475), 980-990. · Zbl 1120.62326
[39] Ledoux, M. & Talagrand, M. (2013) Probability in Banach Spaces: Isoperimetry and Processes. Springer Science & Business Media.
[40] Lee, E.R., Noh, H., & Park, B.U. (2014) Model selection via Bayesian information criterion for quantile regression models. Journal of the American Statistical Association109(505), 216-229. · Zbl 1367.62122
[41] Lee, S., Liao, Y., Seo, M.H., & Shin, Y. (2018) Oracle estimation of a change point in high dimensional quantile regression. Journal of the American Statistical Association43, 1184-1194. · Zbl 1402.62033
[42] Li, Y.J. & Zhu, J. (2008) L1-norm quantile regression. Journal of Computational and Graphical Statistics17, 163-185.
[43] Linton, O.B. & Whang, Y.-J. (2004). A quantilogram approach to evaluating directional predictability. Available at SSRN 485342.
[44] Loh, P.-L. (2017). Statistical consistency and asymptotic normality for high-dimensional robust \(m\) -estimators. Annals of Statistics45(2), 866-896. · Zbl 1371.62023
[45] Loh, P.-L. & Wainwright, M.J. (2012) High-dimensional regression with noisy and missing data: Provable guarantees with nonconvexity. Annals of Statistics40(3), 1637-1664. · Zbl 1257.62063
[46] Loh, P.-L. and Wainwright, M.J. (2015). Regularized \(m\) -estimators with nonconvexity: Statistical and algorithmic theory for local optima. Journal of Machine Learning Research16, 559-616. · Zbl 1360.62276
[47] Lv, S., Lin, H., Lian, H., & Huang, J. (2018) Oracle inequalities for sparse additive quantile regression in reproducing kernel Hilbert space. Annals of Statistics46(2), 781-813. · Zbl 1459.62053
[48] Mei, S., Bai, Y., & Montanari, A. (2018) The landscape of empirical risk for nonconvex losses. Annals of Statistics46(6A), 2747-2774. · Zbl 1409.62117
[49] Negahban, S.N., Ravikumar, P., Wainwright, M.J., & Yu, B. (2012) A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers. Statistical Science27(4), 538-557. · Zbl 1331.62350
[50] Nolan, J. (2003) Stable Distributions: Models for Heavy-Tailed Data. Birkhauser.
[51] Park, S., He, X., & Zhou, S. (2017) Dantzig-type penalization for multiple quantile regression with high dimensional covariates. Statistica Sinica27, 1619-1638. · Zbl 1392.62218
[52] Raskutti, G., Wainwright, M.J., & Yu, B. (2011) Minimax rates of estimation for high-dimensional linear regression over \({l}_q\) -balls. IEEE Transactions on Information Theory57(10), 6976-6994. · Zbl 1365.62276
[53] Ruppert, D. & Carroll, R.J. (1980) Trimmed least squares estimation in the linear model. Journal of the American Statistical Association75(372), 828-838. · Zbl 0459.62055
[54] Sherwood, B. & Wang, L. (2016) Partially linear additive quantile regression in ultra-high dimension. Annals of Statistics44(1), 288-317. · Zbl 1331.62264
[55] Shows, J.H., Lu, W., & Zhang, H.H. (2010) Sparse estimation and inference for censored median regression. Journal of Statistical Planning and Inference140, 1903-1917. · Zbl 1184.62172
[56] Su, L. & Hoshino, T. (2016) Sieve instrumental variable quantile regression estimation of functional coefficient models. Journal of Econometrics191(1), 231-254. · Zbl 1390.62049
[57] Tang, Y., Song, X., Wang, H.J., & Zhu, Z. (2013) Variable selection in high-dimensional quantile varying coefficient models. Journal of Multivariate Analysis122, 115-132. · Zbl 1279.62049
[58] Tao, P.D. & An, L. (1997) Convex analysis approach to D.C. programming: Theory, algorithms and applications. Acta Mathematica Vietnamica22(1), 289-355. · Zbl 0895.90152
[59] Tibshirani, R. (1996) Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society. Series B58, 267-288. · Zbl 0850.62538
[60] Van De Geer, S.A. (2000) Empirical Processes in M-Estimation. Cambridge University Press. · Zbl 1179.62073
[61] Van De Geer, S.A. (2016) Estimation and Testing under Sparsity. Springer. · Zbl 1362.62006
[62] Van Der Vaart, A. & Wellner, J. (1996) Weak Convergence and Empirical Processes: With Applications to Statistics. Springer Science & Business Media. · Zbl 0862.60002
[63] Wagener, J., Volgushev, S., & Dette, H. (2012) The quantile process under random censoring. Mathematical Methods of Statistics21, 127-141. · Zbl 1325.62186
[64] Wang, H., Li, G., & Jiang, G. (2007) Robust regression shrinkage and consistent variable selection through the LAD-lasso. Journal of Business & Economic Statistics25, 347-355.
[65] Wang, H., Zhou, J., & Li, Y. (2013a) Variable selection for censored quantile regression. Statistica Sinica23, 145-167. · Zbl 1257.62046
[66] Wang, L. (2013) The L1 penalized LAD estimator for high dimensional linear regression. Journal of Multivariate Analysis120, 135-151. · Zbl 1279.62144
[67] Wang, L. (2019). L_1-regularized quantile regression with many regressors under lean assumptions. University of Minnesota Digital Conservancy. Available at https://hdl.handle.net/11299/202063.
[68] Wang, L., Kim, Y., & Li, R. (2013b) Calibrating non-convex penalized regression in ultra-high dimension. Annals of Statistics41(5), 2505-2536. · Zbl 1281.62106
[69] Wang, L., Peng, B., Bradic, J., Li, R., & Wu, Y. (2020) A tuning-free robust and efficient approach to high-dimensional regression. Journal of the American Statistical Association115(532), 1700-1714. · Zbl 1452.62525
[70] Wang, L., Wu, Y., & Li, R. (2012) Quantile regression for analyzing heterogeneity in ultra-high dimension. Journal of the American Statistical Association107(497), 214-222. · Zbl 1328.62468
[71] Wu, Y.C. & Liu, Y.F. (2009) Variable selection in quantile regression. Statistica Sinica19, 801-817. · Zbl 1166.62012
[72] Zhang, C.H. (2010) Nearly unbiased variable selection under minimax concave penalty. Annals of Statistics38, 894-942. · Zbl 1183.62120
[73] Zhao, T., Kolar, M., & Liu, H. (2014) A general framework for robust testing and confidence regions in high-dimensional quantile regression. Preprint, arXiv:1412.8724.
[74] Zheng, Q., Peng, L., & He, X. (2015) Globally adaptive quantile regression with ultra-high dimensional data. Annals of Statistics43(5), 2225-2258. · Zbl 1327.62424
[75] Zhong, W., Zhu, L., Li, R., & Cui, H. (2016) Regularized quantile regression and robust feature screening for single index models. Statistica Sinica26(1), 69-95. · Zbl 1419.62096
[76] Zou, H. & Yuan, M. (2008) Composite quantile regression and the oracle model selection theory. Annals of Statistics36, 1108-1126. · Zbl 1360.62394
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.