×

Variable selection and estimation for semi-parametric multiple-index models. (English) Zbl 1388.62116

Summary: We propose a novel method to select significant variables and estimate the corresponding coefficients in multiple-index models with a group structure. All existing approaches for single-index models cannot be extended directly to handle this issue with several indices. This method integrates a popularly used shrinkage penalty such as LASSO with the group-wise minimum average variance estimation. It is capable of simultaneous dimension reduction and variable selection, while incorporating the group structure in predictors. Interestingly, the proposed estimator with the LASSO penalty then behaves like an estimator with an adaptive LASSO penalty. The estimator achieves consistency of variable selection without sacrificing the root-\(n\) consistency of basis estimation. Simulation studies and a real-data example illustrate the effectiveness and efficiency of the new method.

MSC:

62G08 Nonparametric regression and quantile regression
62J07 Ridge regression; shrinkage estimators (Lasso)
62G20 Asymptotic properties of nonparametric inference

Software:

glmnet

References:

[1] Bondell, H.D. and Li, L. (2009). Shrinkage inverse regression estimation for model-free variable selection. J. R. Stat. Soc. Ser. B Stat. Methodol. 71 287-299. · Zbl 05691142 · doi:10.1111/j.1467-9868.2008.00686.x
[2] Breheny, P. and Huang, J. (2011). Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. Ann. Appl. Stat. 5 232-253. · Zbl 1220.62095 · doi:10.1214/10-AOAS388
[3] Chatterjee, A. and Lahiri, S.N. (2010). Asymptotic properties of the residual bootstrap for Lasso estimators. Proc. Amer. Math. Soc. 138 4497-4509. · Zbl 1203.62014 · doi:10.1090/S0002-9939-2010-10474-4
[4] Chatterjee, A. and Lahiri, S.N. (2011). Bootstrapping lasso estimators. J. Amer. Statist. Assoc. 106 608-625. · Zbl 1232.62088 · doi:10.1198/jasa.2011.tm10159
[5] Chen, X., Zou, C. and Cook, R.D. (2010). Coordinate-independent sparse sufficient dimension reduction and variable selection. Ann. Statist. 38 3696-3723. · Zbl 1204.62107 · doi:10.1214/10-AOS826
[6] Cook, R.D. (1998). Regression Graphics : Ideas for Studying Regressions Through Graphics. Wiley Series in Probability and Statistics : Probability and Statistics . New York: Wiley. · Zbl 0903.62001
[7] Cook, R.D. (2004). Testing predictor contributions in sufficient dimension reduction. Ann. Statist. 32 1062-1092. · Zbl 1092.62046 · doi:10.1214/009053604000000292
[8] Cook, R.D. and Li, B. (2002). Dimension reduction for conditional mean in regression. Ann. Statist. 30 455-474. · Zbl 1012.62035 · doi:10.1214/aos/1021379861
[9] Cook, R.D. and Ni, L. (2005). Sufficient dimension reduction via inverse regression: A minimum discrepancy approach. J. Amer. Statist. Assoc. 100 410-428. · Zbl 1117.62312 · doi:10.1198/016214504000001501
[10] Cook, R.D. and Weisberg, S. (1991). Discussion of “Sliced inverse regression for dimension reduction” by K.C. Li. J. Amer. Statist. Assoc. 86 328-332. · Zbl 1353.62037 · doi:10.1080/01621459.1991.10475036
[11] Efron, B., Hastie, T., Johnstone, I. and Tibshirani, R. (2004). Least angle regression. Ann. Statist. 32 407-499. With discussion, and a rejoinder by the authors. · Zbl 1091.62054 · doi:10.1214/009053604000000067
[12] Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Statist. Assoc. 96 1348-1360. · Zbl 1073.62547 · doi:10.1198/016214501753382273
[13] Friedman, J.H., Hastie, T. and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. J. Statist. Software 33 1-22.
[14] Härdle, W. and Stoker, T.M. (1989). Investigating smooth multiple regression by the method of average derivatives. J. Amer. Statist. Assoc. 84 986-995. · Zbl 0703.62052 · doi:10.2307/2290074
[15] Hirst, J.D., King, R.D. and Sternberg, M.J.E. (1994). Quantitative structure-activity relationships by neural networks and inductive logic programming. I. The inhibition of dihydrofolate reductase by pyrimidines. J. Computer-Aided Molecular Design 8 405-420.
[16] Knight, K. and Fu, W. (2000). Asymptotics for lasso-type estimators. Ann. Statist. 28 1356-1378. · Zbl 1105.62357 · doi:10.1214/aos/1015957397
[17] Li, K.-C. (1991). Sliced inverse regression for dimension reduction. J. Amer. Statist. Assoc. 86 316-342. With discussion and a rejoinder by the author. · Zbl 0742.62044 · doi:10.2307/2290563
[18] Li, L., Cook, R.D. and Nachtsheim, C.J. (2005). Model-free variable selection. J. R. Stat. Soc. Ser. B Stat. Methodol. 67 285-299. · Zbl 1069.62053 · doi:10.1111/j.1467-9868.2005.00502.x
[19] Li, L., Li, B. and Zhu, L.-X. (2010). Groupwise dimension reduction. J. Amer. Statist. Assoc. 105 1188-1201. · Zbl 1390.62064 · doi:10.1198/jasa.2010.tm09643
[20] Liang, H., Liu, X., Li, R. and Tsai, C.-L. (2010). Estimation and testing for partially linear single-index models. Ann. Statist. 38 3811-3836. · Zbl 1204.62068 · doi:10.1214/10-AOS835
[21] Peng, H. and Huang, T. (2011). Penalized least squares for single index models. J. Statist. Plann. Inference 141 1362-1379. · Zbl 1204.62070 · doi:10.1016/j.jspi.2010.10.003
[22] Radchenko, P. (2008). Mixed-rates asymptotics. Ann. Statist. 36 287-309. · Zbl 1131.62017 · doi:10.1214/009053607000000668
[23] Schwarz, G. (1978). Estimating the dimension of a model. Ann. Statist. 6 461-464. · Zbl 0379.62005 · doi:10.1214/aos/1176344136
[24] So, S.-S. (2000). Quantitative structure-activity relationships. In Evolutionary Algorithms in Molecular Design (D.E. Clark, ed.) 71-97. Weinheim: Wiley-VCH.
[25] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 58 267-288. · Zbl 0850.62538
[26] Wang, H. and Leng, C. (2007). Unified LASSO estimation by least squares approximation. J. Amer. Statist. Assoc. 102 1039-1048. · Zbl 1306.62167 · doi:10.1198/016214507000000509
[27] Wang, H., Li, R. and Tsai, C.-L. (2007). Tuning parameter selectors for the smoothly clipped absolute deviation method. Biometrika 94 553-568. · Zbl 1135.62058 · doi:10.1093/biomet/asm053
[28] Wang, H. and Xia, Y. (2008). Sliced regression for dimension reduction. J. Amer. Statist. Assoc. 103 811-821. · Zbl 1306.62168 · doi:10.1198/016214508000000418
[29] Wang, Q. and Yin, X. (2008). A nonlinear multi-dimensional variable selection method for high dimensional data: Sparse MAVE. Comput. Statist. Data Anal. 52 4512-4520. · Zbl 1452.62136
[30] Wang, T., Xu, P. and Zhu, L. (2013). Penalized minimum average variance estimation. Statist. Sinica 23 543-569. · Zbl 1379.62049
[31] Wang, T., Xu, P.-R. and Zhu, L.-X. (2012). Non-convex penalized estimation in high-dimensional models with single-index structure. J. Multivariate Anal. 109 221-235. · Zbl 1241.62097 · doi:10.1016/j.jmva.2012.03.009
[32] Xia, Y. (2008). A multiple-index model and dimension reduction. J. Amer. Statist. Assoc. 103 1631-1640. · Zbl 1286.62021 · doi:10.1198/016214508000000805
[33] Xia, Y. and Härdle, W. (2006). Semi-parametric estimation of partially linear single-index models. J. Multivariate Anal. 97 1162-1184. · Zbl 1089.62050 · doi:10.1016/j.jmva.2005.11.005
[34] Xia, Y., Tong, H., Li, W.K. and Zhu, L.-X. (2002). An adaptive estimation of dimension reduction space. J. R. Stat. Soc. Ser. B Stat. Methodol. 64 363-410. · Zbl 1091.62028 · doi:10.1111/1467-9868.03411
[35] Xia, Y., Zhang, D. and Xu, J. (2010). Dimension reduction and semiparametric estimation of survival models. J. Amer. Statist. Assoc. 105 278-290. · Zbl 1397.62377 · doi:10.1198/jasa.2009.tm09372
[36] Ye, Z. and Weiss, R.E. (2003). Using the bootstrap to select one of a new class of dimension reduction methods. J. Amer. Statist. Assoc. 98 968-979. · Zbl 1045.62034 · doi:10.1198/016214503000000927
[37] Yin, X. and Li, B. (2011). Sufficient dimension reduction based on an ensemble of minimum average variance estimators. Ann. Statist. 39 3392-3416. · Zbl 1246.62141 · doi:10.1214/11-AOS950
[38] Yin, X., Li, B. and Cook, R.D. (2008). Successive direction extraction for estimating the central subspace in a multiple-index regression. J. Multivariate Anal. 99 1733-1757. · Zbl 1144.62030 · doi:10.1016/j.jmva.2008.01.006
[39] Zeng, P., He, T. and Zhu, Y. (2012). A lasso-type approach for estimation and variable selection in single index models. J. Comput. Graph. Statist. 21 92-109. · doi:10.1198/jcgs.2011.09156
[40] Zhang, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. Ann. Statist. 38 894-942. · Zbl 1183.62120 · doi:10.1214/09-AOS729
[41] Zhu, L.-P., Li, L., Li, R. and Zhu, L.-X. (2011). Model-free feature screening for ultrahigh-dimensional data. J. Amer. Statist. Assoc. 106 1464-1475. · Zbl 1233.62195 · doi:10.1198/jasa.2011.tm10563
[42] Zou, H. (2006). The adaptive lasso and its oracle properties. J. Amer. Statist. Assoc. 101 1418-1429. · Zbl 1171.62326 · doi:10.1198/016214506000000735
[43] Zou, H., Hastie, T. and Tibshirani, R. (2007). On the “degrees of freedom” of the lasso. Ann. Statist. 35 2173-2192. · Zbl 1126.62061 · doi:10.1214/009053607000000127
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.