Large-scale regression with non-convex loss and penalty. (English) Zbl 1445.62161
Summary: We describe a computational method for parameter estimation in linear regression, that is capable of simultaneously producing sparse estimates and dealing with outliers and heavy-tailed error distributions. The method used is based on the image restoration method proposed in [G. Huang et al., BIT 57, No. 2, 351–378 (2017; Zbl 1369.65073)]. It can be applied to problems of arbitrary size. The choice of certain parameters is discussed. Results obtained for simulated and real data are presented.
MSC:
| 62J05 | Linear regression; mixed models |
| 62F10 | Point estimation |
| 62F35 | Robustness and adaptive procedures (parametric inference) |
| 90C26 | Nonconvex programming, global optimization |
| 65K05 | Numerical mathematical programming methods |
| 65D18 | Numerical aspects of computer graphics, image analysis, and computational geometry |
| 94A08 | Image processing (compression, reconstruction, etc.) in information and communication theory |
Citations:
Zbl 1369.65073References:
| [1] | Azzalini, A.; Capitanio, A., The Skew-Normal and Related Families (2014), Cambridge University Press: Cambridge University Press Cambridge · Zbl 0924.62050 |
| [2] | Beck, A.; Teboulle, M., A fast iterative shrinkage-thresholding algorithm for linear inverse problems, SIAM J. Imaging Sci., 2, 183-202 (2009) · Zbl 1175.94009 |
| [3] | Buccini, A.; Pasha, M.; Reichel, L., Modulus-based iterative methods for constrained \(\ell_p- \ell_q\) minimization, Inverse Probl. (2020), in press · Zbl 1451.65072 |
| [4] | Buccini, A.; Reichel, L., An \(\ell^2- \ell^q\) regularization method for large discrete ill-posed problems, J. Sci. Comput., 78, 1526-1549 (2019) · Zbl 1502.65017 |
| [5] | Buccini, A.; Reichel, L., An \(\ell_p- \ell_q\) minimization method with cross-validation for the restoration of impulse noise contaminated images, J. Comput. Appl. Math., 375, Article 112824 pp. (2020) · Zbl 1524.94008 |
| [6] | Candès, E. J.; Romberg, J.; Tao, T., Robust uncertainty principle: exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inf. Theory, 52, 489-509 (2006) · Zbl 1231.94017 |
| [7] | Candès, E. J.; Wakin, M. B.; Boyd, S. P., Enhancing sparsity by reweighted \(\ell_1\) minimization, J. Fourier Anal. Appl., 14, 877-905 (2008) · Zbl 1176.94014 |
| [8] | Chan, R. H.; Liang, H. X., Half-quadratic algorithm for \(\ell_p- \ell_q\) problems with applications to TV-\( \ell_1\) image restoration and compressive sensing, (Efficient Algorithms for Global Optimization Methods in Computer Vision. Efficient Algorithms for Global Optimization Methods in Computer Vision, Lecture Notes in Computer Science, vol. 8293 (2014), Springer: Springer Berlin), 78-103 |
| [9] | Donoho, D., Compressed sensing, IEEE Trans. Inf. Theory, 52, 1289-1306 (2006) · Zbl 1288.94016 |
| [10] | Engl, H. W.; Hanke, M.; Neubauer, A., Regularization of Inverse Problems (1996), Kluwer: Kluwer Dordrecht · Zbl 0859.65054 |
| [11] | Hoerl, A. E.; Kennard, R. W., Ridge regression: biased estimation for nonorthogonal problems, Technometrics, 12, 55-67 (1970) · Zbl 0202.17205 |
| [12] | Horst, R.; Thoai, N. V., DC programming: overview, J. Optim. Theory Appl., 103, 1-43 (1999) · Zbl 1073.90537 |
| [13] | Huang, G.; Lanza, A.; Morigi, S.; Reichel, L.; Sgallari, F., Majorization-minimization generalized Krylov subspace methods for \(\ell_p- \ell_q\) optimization applied to image restoration, BIT Numer. Math., 57, 351-378 (2017) · Zbl 1369.65073 |
| [14] | Huber, P. J.; Ronchetti, E. M., Robust Statistics (2009), Wiley: Wiley Hoboken · Zbl 1276.62022 |
| [15] | Ghazalpour, A.; Bennett, B. J.; Shih, D., Genetic regulation of mouse liver metabolite levels, Mol. Syst. Biol., 10, 5, 730 (2014) |
| [16] | Lampe, J.; Reichel, L.; Voss, H., Large-scale Tikhonov regularization via reduction by orthogonal projection, Linear Algebra Appl., 436, 2845-2865 (2012) · Zbl 1241.65044 |
| [17] | Lanza, A.; Morigi, S.; Reichel, L.; Sgallari, F., A generalized Krylov subspace method for \(\ell_p- \ell_q\) minimization, SIAM J. Sci. Comput., 37, S30-S50 (2015) · Zbl 1343.65077 |
| [18] | Lanza, A.; Morigi, S.; Sgallari, F., Constrained TV_p-\( \ell_2\) model for image restoration, J. Sci. Comput., 68, 64-91 (2016) · Zbl 1344.65025 |
| [19] | Liu, Z.; Wei, Z.; Sun, W., An iteratively approximated gradient projection algorithm for sparse signal reconstruction, Appl. Math. Comput., 228, 454-462 (2014) · Zbl 1364.94141 |
| [20] | Mazumder, R.; Friedman, J. H.; Hastie, T., Sparsenet: coordinate descent with nonconvex penalties, J. Am. Stat. Assoc., 106, 1125-1138 (2011) · Zbl 1229.62091 |
| [21] | Nikolova, M.; Chan, R. H., The equivalence of the half-quadratic minimization and the gradient linearization iteration, IEEE Trans. Image Process., 16, 5-18 (2007) |
| [22] | Ramlau, R.; Zarzer, C. A., On the minimization of a Tikhonov functional with a non-convex sparsity constraint, Electron. Trans. Numer. Anal., 39, 476-507 (2012) · Zbl 1287.65044 |
| [23] | Rodríguez, P.; Wohlberg, B., Efficient minimization method for a generalized total variation functional, IEEE Trans. Image Process., 18, 322-332 (2009) · Zbl 1371.94316 |
| [24] | Rodríguez, P.; Wohlberg, B., Numerical methods for inverse problems and adaptive decomposition (NUMIPAD), software library available from |
| [25] | Rudin, L.; Osher, S.; Fatemi, E., Nonlinear total variation based noise removal algorithms, Physica D, 60, 259-268 (1992) · Zbl 0780.49028 |
| [26] | Stone, M., Cross-validatory choice and assessment of statistical prediction, J. R. Stat. Soc. Ser. B, 36, 111-147 (1977) · Zbl 0308.62063 |
| [27] | Subbotin, M. Th., On the law of frequency of error, Mat. Sb., 31, 296-301 (1923) · JFM 49.0370.01 |
| [28] | Tibshirani, R., Regression shrinkage and selection via the Lasso, J. R. Stat. Soc. Ser. B, 58, 267-288 (1996) · Zbl 0850.62538 |
| [29] | Tibshirani, R.; Hastie, T.; Friedman, J., The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2009), Springer: Springer New York · Zbl 1273.62005 |
| [30] | Weisberg, S., Applied Linear Regression (2014), Wiley: Wiley Hoboken · Zbl 1281.62015 |
| [31] | Wolke, R.; Schwetlick, H., Iteratively reweighted least squares: algorithms, convergence analysis, and numerical comparisons, SIAM J. Sci. Stat. Comput., 9, 907-921 (1988) · Zbl 0709.65130 |
| [32] | Yuille, A. L.; Rangarajan, A., The convex-concave procedure, Neural Comput., 15, 915-936 (2003) · Zbl 1022.68112 |
| [33] | Zhao, Y.; Li, D., Reweighted \(\ell_1\)-minimization for sparse solutions to undetermined linear systems, SIAM J. Optim., 22, 1065-1088 (2012) · Zbl 1261.65042 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
