Shrinkage and model selection with correlated variables via weighted fusion. (English) Zbl 1452.62049
Summary: We propose the weighted fusion, a new penalized regression and variable selection method for data with correlated variables. The weighted fusion can potentially incorporate information redundancy among correlated variables for estimation and variable selection. Weighted fusion is also useful when the number of predictors \(p\) is larger than the number of observations \(n\). It allows the selection of more than \(n\) variables in a motivated way. Real data and simulation examples show that weighted fusion can improve variable selection and prediction accuracy.
MSC:
| 62-08 | Computational methods for problems pertaining to statistics |
| 62J05 | Linear regression; mixed models |
| 62J07 | Ridge regression; shrinkage estimators (Lasso) |
| 62H12 | Estimation in multivariate analysis |
References:
| [1] | Bickel, P.J., Levina, E., Covariance regularization by thresholding, Ann. Statist. (in press) 2008; Bickel, P.J., Levina, E., Covariance regularization by thresholding, Ann. Statist. (in press) 2008 · Zbl 1196.62062 |
| [2] | Bickel, P. J.; Levina, E., Regularized estimation of large covariance matrices, Ann. Statist., 36, 199-227 (2008) · Zbl 1132.62040 |
| [3] | Bøvelstad, H. M.; Nygård, S.; Størvold, H. L.; Aldrin, M.; Borgan, Ø.; Frigessi, A.; Lingjærde, O. C., Predicting survival from microarray data-a comparative study, Bioinformatics, 23, 16, 2080-2087 (2007) |
| [4] | Bühlmann, P., Boosting for high-dimensional linear models, Ann. Statist., 34, 559-583 (2006) · Zbl 1095.62077 |
| [5] | Bühlmann, P.; Yu, B., Boosting with the l2 loss: Regression and classification, J. Amer. Statist. Assos., 98, 324-339 (2003) · Zbl 1041.62029 |
| [6] | Donoho, D. L.; Johnstone, I. M., Ideal spatial adaptation by wavelet shrinkage, Biometrika, 81, 425-455 (1994) · Zbl 0815.62019 |
| [7] | Donoho, D. L.; Johnstone, I. M.; Kerkyacharian, G.; Picard, D., Wavelet shrinkage; asymptopia?, J. Roy. Statist. Soc. B, 57, 301-337 (1995) · Zbl 0827.62035 |
| [8] | Efron, B.; Hastie, T.; Johnstone, I.; Tibshirani, R., Least angle regression, Ann. Statist., 32, 407-499 (2004) · Zbl 1091.62054 |
| [9] | El Karoui, N., 2007, Operator norm consistent estimation of large dimensional sparse covariance matrices, Ann. Statist. (in press); El Karoui, N., 2007, Operator norm consistent estimation of large dimensional sparse covariance matrices, Ann. Statist. (in press) · Zbl 1196.62064 |
| [10] | Fan, J.; Li, R., Statistical challenges with high dimensionality: Feature selection in knowledge discovery, (Sanz-Sole, M.; Soria, J.; Varona, J. L.; Verdera, J., Proceedings of the International Congress of Mathematicians, vol. 3 (2006), European Mathematical Society: European Mathematical Society Zurich), 595-622 · Zbl 1117.62137 |
| [11] | Harrell, F. E., Regression modeling strategies (2001), Springer · Zbl 0982.62063 |
| [12] | Hoerl, A. E.; Kennard, R. W., Ridge regression: Applications to nonorthogonal problems, Technometrics, 12, 69-82 (1970) · Zbl 0202.17206 |
| [13] | Hoerl, A. E.; Kennard, R. W., Ridge regression: Biased estimation for nonorthogonal problems, Technometrics, 12, 55-67 (1970) · Zbl 0202.17205 |
| [14] | Huang, J.; Liu, N.; Pourahmadi, M.; Liu, L., Covariance matrix selection and estimation via penalized normal likelihood, Biometrika, 93(1), 85-98 (2006) · Zbl 1152.62346 |
| [15] | Hurvich, C. M.; Simonoff, J. S.; Tsai, C., Smoothing parameter selection in nonparametric regression using an improved akaike information criterion, J. R. Statist. Soc. B, 60, 271-293 (1998) · Zbl 0909.62039 |
| [16] | Kim, Y., Kim, J., 2004. Gradient lasso for feature selection, Proceedings of the 21st International Conference on Machine Learning; Kim, Y., Kim, J., 2004. Gradient lasso for feature selection, Proceedings of the 21st International Conference on Machine Learning |
| [17] | Knight, K.; Fu, W. J., Asymptotics for lasso-type estimators, Ann. Statist., 28, 1356-1378 (2000) · Zbl 1105.62357 |
| [18] | Kutner, Michael H.; Nachtsheim, Cristopher J.; Neter, John; Li, William, Applied Linear Statistical Models (2005), McGraw-Hill/Irwin |
| [19] | Lam, C., Fan, J., 2007. Sparsistency and rates of convergence in large covariance matrices estimation, Manuscript, Department of Operations Research and Financial Engineering, Princeton University; Lam, C., Fan, J., 2007. Sparsistency and rates of convergence in large covariance matrices estimation, Manuscript, Department of Operations Research and Financial Engineering, Princeton University |
| [20] | Land, S., Friedman, J., 1996. Variable fusion: A new method of adaptive signal regression. Technical report, Department of Statistics, Stanford University; Land, S., Friedman, J., 1996. Variable fusion: A new method of adaptive signal regression. Technical report, Department of Statistics, Stanford University |
| [21] | Li, C.; Li, H., Network-constrained regularization and variable selection for analysis of genomic data, Bioinformatics (Advance Access) (2008) |
| [22] | Mosteller, F.; Tukey, J. W., Data analysis and regression: A second course in statistics (1977), Addison-Wesley |
| [23] | Park, M. Y.; Hastie, T.; Tibshirani, R., Averaged gene expressions for regression, Biostatistics, 8, 212-227 (2007) · Zbl 1144.62357 |
| [24] | Rosset, S.; Zhu, J., Piecewise linear regularized solution, Ann. Statist., 35, 1012-1030 (2007) · Zbl 1194.62094 |
| [25] | Rothman, A.J., Bickel, P.J., Levina, E., Zhu, J., 2007. Sparse permutation invariant covariance estimation. Technical Report 467, Department of Statistics, University of Michigan; Rothman, A.J., Bickel, P.J., Levina, E., Zhu, J., 2007. Sparse permutation invariant covariance estimation. Technical Report 467, Department of Statistics, University of Michigan · Zbl 1320.62135 |
| [26] | Rothman, A.J., Levina, E., Zhu, J., 2008. Generalized thresholding of large covariance matrices. Technical Report 477, Department of Statistics, University of Michigan; Rothman, A.J., Levina, E., Zhu, J., 2008. Generalized thresholding of large covariance matrices. Technical Report 477, Department of Statistics, University of Michigan · Zbl 1388.62170 |
| [27] | Segal, M.; Dahlquist, K.; Conklin, B., Regression approach for microarray data analysis, Journal of Computational Biology, 10, 961-980 (2003) |
| [28] | Tibshirani, R., Regression shrinkage and selection via the lasso, J. Roy. Statist. Soc. B, 58, 267-288 (1996) · Zbl 0850.62538 |
| [29] | Tibshirani, R.; Saunders, M.; Rosset, S.; Zhu, J.; Knight, K., Sparsity and smoothness via the fused lasso, J. R. Statist. Soc. B, 67, 91-108 (2005) · Zbl 1060.62049 |
| [30] | Tutz, G., Ulbricht, J., 2006. Penalized regression with correlation based penalty. Discussion paper 486, SFB 386, Universität München; Tutz, G., Ulbricht, J., 2006. Penalized regression with correlation based penalty. Discussion paper 486, SFB 386, Universität München |
| [31] | Wang, L.; Zhu, J.; Zou, H., The doubly regularized support vector machine, Statist. Sinica, 16, 589-615 (2006) · Zbl 1126.68070 |
| [32] | Weisberg, Sanford, Applied Linear Regression (1985), John Wiley & Sons · Zbl 0646.62058 |
| [33] | Wong, F.; Carter, C.; Kohn, R., Efficient estimation of covariance selection models, Biometrika, 90, 809-830 (2003) · Zbl 1436.62346 |
| [34] | Wu, W. B.; Pourahmadi, M., Nonparametric estimation of large covariance matrices of longitudinal data, Biometrika, 90, 831-844 (2003) · Zbl 1436.62347 |
| [35] | Yu, B., Embracing statistical challenges in the information technology age, Technometrics, 49, 237-248 (2007) |
| [36] | Yuan, M.; Lin, Y., Model selection and estimation in regression with grouped variables, J. R. Statist. Soc. B, 68, 49-67 (2006) · Zbl 1141.62030 |
| [37] | Yuan, M.; Lin, Y., On the non-negative garrotte estimator, J. Roy. Statist. Soc. B, 69, 143-161 (2007) · Zbl 1120.62052 |
| [38] | Zhao, P.; Yu, B., On model selection consistency of lasso, Journal of Machine Learning Research, 7, 2541-2567 (2006) · Zbl 1222.62008 |
| [39] | Zhao, P.; Yu, B., Stagewise lasso, Journal of Machine Learning Research, 8, 2701-2726 (2007) · Zbl 1222.68345 |
| [40] | Zou, H., The adaptive lasso and its oracle properties, J. Amer. Statist. Assoc., 101, 1418-1429 (2006) · Zbl 1171.62326 |
| [41] | Zou, H.; Hastie, T., Regularization and variable selection via the elastic net, J. R. Statist. Soc. B, 67, 301-320 (2005) · Zbl 1069.62054 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
