Tuning parameter selection in sparse regression modeling. (English) Zbl 1400.62006
Summary: In sparse regression modeling via regularization such as the lasso, it is important to select appropriate values of tuning parameters including regularization parameters. The choice of tuning parameters can be viewed as a model selection and evaluation problem. Mallows’ \(C_p\) type criteria may be used as a tuning parameter selection tool in lasso type regularization methods, for which the concept of degrees of freedom plays a key role. In the present paper, we propose an efficient algorithm that computes the degrees of freedom by extending the generalized path seeking algorithm. Our procedure allows us to construct model selection criteria for evaluating models estimated by regularization with a wide variety of convex and nonconvex penalties. The proposed methodology is investigated through the analysis of real data and Monte Carlo simulations. Numerical results show that \(C_p\) criterion based on our algorithm performs well in various situations.
MSC:
| 62-07 | Data analysis (statistics) (MSC2010) |
| 62F07 | Statistical ranking and selection procedures |
| 62J07 | Ridge regression; shrinkage estimators (Lasso) |
Keywords:
degrees of freedom; generalized path seeking; model selection; regularization; sparse regressionReferences:
| [1] | Akaike, H., Information theory and an extension of the maximum likelihood principle, (Petrov, B. N.; Csaki, F., 2nd International Symposium on Information Theory, (1973), Akademiai Kiado Budapest), 267-281 · Zbl 0283.62006 |
| [2] | Baudry, J.-P.; Maugis, C.; Michel, B., Slope heuristics: overview and implementation, Statistics and Computing, 22, 2, 455-470, (2011) · Zbl 1322.62007 |
| [3] | Breiman, L., Heuristics of instability and stabilization in model selection, The Annals of Statistics, 24, 2350-2383, (1996) · Zbl 0867.62055 |
| [4] | Candès, E.; Wakin, M.; Boyd, S., Enhancing sparsity by reweighted \(\ell_1\) minimization, The Journal of Fourier Analysis and Applications, 14, 877-905, (2008) · Zbl 1176.94014 |
| [5] | Craven, P.; Wahba, G., Smoothing noisy data with spline functions: estimating the correct degree of smoothing by the method of generalized cross-validation, Numerische Mathematik, 31, 377-403, (1979) · Zbl 0377.65007 |
| [6] | Efron, B., How biased is the apparent error rate of a prediction rule ?, Journal of the American Statistical Association, 81, 461-470, (1986) · Zbl 0621.62073 |
| [7] | Efron, B., The estimation of prediction error: covariance penalties and cross-validation, Journal of the American Statistical Association, 99, 619-642, (2004) · Zbl 1117.62324 |
| [8] | Efron, B.; Hastie, T.; Johnstone, I.; Tibshirani, R., Least angle regression (with discussion), The Annals of Statistics, 32, 407-499, (2004) · Zbl 1091.62054 |
| [9] | Fan, J.; Li, R., Variable selection via nonconcave penalized likelihood and its oracle properties, Journal of the American Statistical Association, 96, 1348-1360, (2001) · Zbl 1073.62547 |
| [10] | Frank, I.; Friedman, J., A statistical view of some chemometrics regression tools, Technometrics, 35, 109-148, (1993) · Zbl 0775.62288 |
| [11] | Friedman, J., 2008. Fast sparse regression and classification. Tech. Rep., Stanford Research Institute, California.; Friedman, J., 2008. Fast sparse regression and classification. Tech. Rep., Stanford Research Institute, California. |
| [12] | Friedman, J.; Hastie, H.; Höfling, H.; Tibshirani, R., Pathwise coordinate optimization, The Annals of Applied Statistics, 1, 302-332, (2007) · Zbl 1378.90064 |
| [13] | Friedman, J.; Hastie, T.; Tibshirani, R., Regularization paths for generalized linear models via coordinate descent, Journal of Statistical Software, 33, (2010) |
| [14] | Fu, W., Penalized regressions: the bridge versus the lasso, Journal of Computational and Graphical Statistics, 7, 397-416, (1998) |
| [15] | Gao, X.; Fang, Y., A note on the generalized degrees of freedom under the l1 loss function, Journal of Statistical Planning and Inference, 141, 2, 677-686, (2011) · Zbl 1209.62003 |
| [16] | Hastie, T.; Tibshirani, R., (Generalized Additive Models, Monographs on Statistics and Applied Probability, (1990), Chapman and Hall, CRC London) · Zbl 0747.62061 |
| [17] | Hastie, T.; Tibshirani, R.; Friedman, J., The elements of statistical learning, (2008), Springer New York |
| [18] | Hirose, H., Soejima, Y., Hirose, K., 2012. NNRMLR: A combined method of nearest neighbor regression and multiple linear regression. In: 6th International Workshop on e-Activity, IWEA 2012.; Hirose, H., Soejima, Y., Hirose, K., 2012. NNRMLR: A combined method of nearest neighbor regression and multiple linear regression. In: 6th International Workshop on e-Activity, IWEA 2012. |
| [19] | Hurvich, C. M.; Simonoff, J. S.; Tsai, C.-L., Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion, Journal of the Royal Statistical Society, Series B, 60, 271-293, (1998) · Zbl 0909.62039 |
| [20] | Kato, K., On the degrees of freedom in shrinkage estimation, Journal of Multivariate Analysis, 100, 1338-1352, (2009) · Zbl 1162.62067 |
| [21] | Konishi, S.; Kitagawa, G., Information criteria and statistical modeling, (2008), Springer New York · Zbl 1172.62003 |
| [22] | Mallows, C., Some comments on \(c_p\), Technometrics, 15, 661-675, (1973) · Zbl 0269.62061 |
| [23] | Mazumder, R.; Friedman, J.; Hastie, T., Sparsenet: coordinate descent with nonconvex penalties, Journal of the American Statistical Association, 106, 1125-1138, (2011) · Zbl 1229.62091 |
| [24] | R Development Core Team, R: A language and environment for statistical computing, (2010), R Foundation for Statistical Computing Vienna, Austria, URL: http://www.R-project.org |
| [25] | Schwarz, G., Estimation of the mean of a multivariate normal distribution, The Annals of Statistics, 9, 1135-1151, (1978) · Zbl 0476.62035 |
| [26] | Shen, X.; Huang, H.-C.; Ye, J., Adaptive model selection and assessment for exponential family distributions, Technometrics, 46, 306-317, (2004) |
| [27] | Shen, X.; Ye, J., Adaptive model selection, Journal of the American Statistical Association, 97, 210-221, (2002) · Zbl 1073.62509 |
| [28] | Stein, C., Estimation of the mean of a multivariate normal distribution, The Annals of Statistics, 9, 1135-1151, (1981) · Zbl 0476.62035 |
| [29] | Sugiura, N., Further analysts of the data by akaike’s information criterion and the finite corrections, Communications in Statistics, Part A—Theory and Methods, 7, 13-26, (1978) · Zbl 0382.62060 |
| [30] | Tibshirani, R., Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society, Series B, 58, 267-288, (1996) · Zbl 0850.62538 |
| [31] | Tibshirani, R.; Taylor, J., The solution path of the generalized lasso, The Annals of Statistics, 39, 3, 1335-1371, (2011) · Zbl 1234.62107 |
| [32] | Wang, H.; Li, B.; Leng, C., Shrinkage tuning parameter selection with a diverging number of parameters, Journal of the Royal Statistical Society, Series B, 71, 671-683, (2009) · Zbl 1250.62036 |
| [33] | Wang, H.; Li, R.; Tsai, C., Tuning parameter selectors for the smoothly clipped absolute deviation method, Biometrika, 94, 553-568, (2007) · Zbl 1135.62058 |
| [34] | Ye, J., On measuring and correcting the effects of data mining and model selection, Journal of the American Statistical Association, 93, 120-131, (1998) · Zbl 0920.62056 |
| [35] | Yuan, M.; Lin, Y., Model selection and estimation in regression with grouped variables, Journal of the Royal Statistical Society, Series B, 68, 49-67, (2006) · Zbl 1141.62030 |
| [36] | Zhang, C., Nearly unbiased variable selection under minimax concave penalty, The Annals of Statistics, 38, 894-942, (2010) · Zbl 1183.62120 |
| [37] | Zhao, P.; Rocha, G.; Yu, B., The composite absolute penalties family for grouped and hierarchical variable selection, The Annals of Statistics, 37, 3468-3497, (2009) · Zbl 1369.62164 |
| [38] | Zou, H., The adaptive lasso and its oracle properties, Journal of the American Statistical Association, 101, 1418-1429, (2006) · Zbl 1171.62326 |
| [39] | Zou, H.; Hastie, T., Regularization and variable selection via the elastic net, Journal of the Royal Statistical Society, Series B, 67, 301-320, (2005) · Zbl 1069.62054 |
| [40] | Zou, H.; Hastie, T.; Tibshirani, R., On the “degrees of freedom” of the lasso, The Annals of Statistics, 35, 2173-2192, (2007) · Zbl 1126.62061 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
