A unifying representer theorem for inverse problems and machine learning. (English) Zbl 1479.46088
A novel higher-level formulation of regularization is proposed in the context of Banach spaces. A general representer theorem for characterizing the solutions of a broad class of optimization problems is proven and then shown to be employable for rediscovering various known similar results from the literature.
Reviewer: Sorin-Mihai Grad (Paris)
MSC:
| 46N10 | Applications of functional analysis in optimization, convex analysis, mathematical programming, economics |
| 47A52 | Linear operators and ill-posed problems, regularization |
| 65J20 | Numerical solutions of ill-posed problems in abstract spaces; regularization |
| 68T05 | Learning and adaptive systems in artificial intelligence |
Keywords:
convex optimization; regularization; representer theorem; inverse problem; machine learning; Banach spaceReferences:
| [1] | Alvarez, MA; Rosasco, L.; Lawrence, ND, Kernels for vector-valued functions: A review, Foundations and Trends in Machine Learning, 4, 3, 195-266 (2012) · Zbl 1301.68212 |
| [2] | Argyriou, A.; Micchelli, CA; Pontil, M., When is there a representer theorem? Vector versus matrix regularizers, Journal of Machine Learning Research, 10, Nov, 2507-2529 (2009) · Zbl 1235.68128 |
| [3] | Aronszajn, N., Theory of reproducing kernels, Transactions of the American Mathematical Society, 68, 3, 337-404 (1950) · Zbl 0037.20701 |
| [4] | Beurling, A.; Livingston, AE, A theorem on duality mappings in Banach spaces, Arkiv för Matematik, 4, 5, 405-411 (1962) · Zbl 0105.09301 |
| [5] | Bohn, B.; Rieger, C.; Griebel, M., A representer theorem for deep kernel learning, Journal of Machine Learning Research, 20, 64, 1-32 (2019) · Zbl 1489.62197 |
| [6] | de Boor, C., On “best” interpolation, Journal of Approximation Theory, 16, 1, 28-42 (1976) · Zbl 0314.41001 |
| [7] | de Boor, C.; Lynch, RE, On splines and their minimum properties, Journal of Mathematics and Mechanics, 15, 6, 953-969 (1966) · Zbl 0185.20501 |
| [8] | Boyer, C.; Chambolle, A.; De Castro, Y.; Duval, V.; De Gournay, F.; Weiss, P., On representer theorems and convex regularization, SIAM Journal of Optimization, 29, 2, 1260-1281 (2019) · Zbl 1423.49036 |
| [9] | Bredies, K.; Carioni, M., Sparsity of solutions for variational inverse problems with finite-dimensional data, Calculus of Variations and Partial Differential Equations, 59, 14, 26 (2020) · Zbl 1430.49036 |
| [10] | Bredies, K.; Pikkarainen, HK, Inverse problems in spaces of measures, ESAIM: Control, Optimisation and Calculus of Variations, 19, 1, 190-218 (2013) · Zbl 1266.65083 |
| [11] | Bruckstein, AM; Donoho, DL; Elad, M., From sparse solutions of systems of equations to sparse modeling of signals and images, SIAM Review, 51, 1, 34-81 (2009) · Zbl 1178.68619 |
| [12] | Candès, EJ; Fernandez-Granda, C., Super-resolution from noisy data, Journal of Fourier Analysis and Applications, 19, 6, 1229-1254 (2013) · Zbl 1312.94015 |
| [13] | Candès, EJ; Fernandez-Granda, C., Towards a mathematical theory of super-resolution, Communications on Pure and Applied Mathematics, 67, 6, 906-956 (2014) · Zbl 1350.94011 |
| [14] | Candès, EJ; Romberg, J., Sparsity and incoherence in compressive sampling, Inverse Problems, 23, 3, 969-985 (2007) · Zbl 1120.94005 |
| [15] | Cioranescu, I.: Geometry of Banach Spaces, Duality Mappings and Nonlinear Problems, vol. 62. Springer Science & Business Media, (2012) · Zbl 0712.47043 |
| [16] | Combettes, PL; Salzo, S.; Villa, S., Regularized learning schemes in feature Banach spaces, Analysis and Applications, 16, 1, 1-54 (2018) · Zbl 1378.62015 |
| [17] | Denoyelle, Q.; Duval, V.; Peyré, G., Support recovery for sparse super-resolution of positive measures, Journal of Fourier Analysis and Applications, 23, 5, 1153-1194 (2017) · Zbl 1417.65223 |
| [18] | Dodu, F.; Rabut, C., Irrotational or divergence-free interpolation, Numerische Mathematik, 98, 477-498 (2004) · Zbl 1140.41302 |
| [19] | Donoho, DL, Compressed sensing, IEEE Transactions on Information Theory, 52, 4, 1289-1306 (2006) · Zbl 1288.94016 |
| [20] | Duchon, J.; Schempp, W.; Zeller, K., Splines minimizing rotation-invariant semi-norms in Sobolev spaces, Constructive Theory of Functions of Several Variables, 85-100 (1977), Berlin: Springer-Verlag, Berlin · Zbl 0342.41012 |
| [21] | Duval, V.; Peyré, G., Exact support recovery for sparse spikes deconvolution, Foundations of Computational Mathematics, 15, 5, 1315-1355 (2015) · Zbl 1327.65104 |
| [22] | Ekeland, I., Temam, R.: Convex Analysis and Variational Problems, Classics in Applied Mathematics, vol. 28. SIAM (1999) · Zbl 0939.49002 |
| [23] | Evgeniou, T.; Pontil, M.; Poggio, T., Regularization networks and support vector machines, Advances in Computational Mathematics, 13, 1, 1-50 (2000) · Zbl 0939.68098 |
| [24] | Fernandez-Granda, C., Super-resolution of point sources via convex programming, Information and Inference, 5, 251-303 (2016) · Zbl 1386.94027 |
| [25] | Fisher, SD; Jerome, JW, Spline solutions to \(L_1\) extremal problems in one and several variables, Journal of Approximation Theory, 13, 1, 73-83 (1975) · Zbl 0295.49002 |
| [26] | Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressive Sensing. Springer (2013) · Zbl 1315.94002 |
| [27] | Gelfand, IM; Vilenkin, NY, Generalized Functions (1964), New York, USA: Applications of Harmonic Analysis. Academic Press, New York, USA · Zbl 0136.11201 |
| [28] | Gupta, H.; Fageot, J.; Unser, M., Continuous-domain solutions of linear inverse problems with Tikhonov versus generalized TV regularization, IEEE Transactions on Signal Processing, 66, 17, 4670-4684 (2018) · Zbl 1415.94119 |
| [29] | Hofmann, T.; Schölkopf, B.; Smola, AJ, Kernel methods in machine learning, Annals of Statistics, 36, 3, 1171-1220 (2008) · Zbl 1151.30007 |
| [30] | Karayiannis, NB; Venetsanopoulos, AN, Regularization theory in image restoration—The stabilizing functional approach, IEEE Transactions on Acoustics, Speech and Signal Processing, 38, 7, 1155-1179 (1990) · Zbl 0713.93059 |
| [31] | Kimeldorf, G.; Wahba, G., Some results on Tchebycheffian spline functions, Journal of Mathematical Analysis and Applications, 33, 1, 82-95 (1971) · Zbl 0201.39702 |
| [32] | Megginson, R.E.: An Introduction to Banach Space Theory. Springer (1998) · Zbl 0910.46008 |
| [33] | Micchelli, CA; Pontil, M., On learning vector-valued functions, Neural Computation, 17, 1, 177-204 (2005) · Zbl 1092.93045 |
| [34] | Mosamam, A.; Kent, J., Semi-reproducing kernel Hilbert spaces, splines and increment kriging, Journal of Nonparametric Statistics, 22, 6, 711-722 (2010) · Zbl 1359.62133 |
| [35] | Poggio, T.; Girosi, F., Networks for approximation and learning, Proceedings of the IEEE, 78, 9, 1481-1497 (1990) · Zbl 1226.92005 |
| [36] | Poggio, T.; Girosi, F., Regularization algorithms for learning that are equivalent to multilayer networks, Science, 247, 4945, 978-982 (1990) · Zbl 1226.92005 |
| [37] | Poon, C.; Peyré, G., Multidimensional sparse super-resolution, SIAM Journal on Mathematical Analysis, 51, 1, 1-44 (2019) · Zbl 1456.94009 |
| [38] | Riesz, M., Sur les fonctions conjuguées, Mathematische Zeitschrift, 27, 218-244 (1927) · JFM 53.0259.02 |
| [39] | Roth, V., The generalized LASSO, IEEE Transactions on Neural Networks, 15, 1, 16-28 (2004) |
| [40] | Rudin, W., Real and Complex Analysis (1987), New York: McGraw-Hill, New York · Zbl 0925.00005 |
| [41] | Rudin, W.: Functional Analysis, 2nd edn. McGraw-Hill, New York (1991). McGraw-Hill Series in Higher Mathematics · Zbl 0867.46001 |
| [42] | Schölkopf, B.; Herbrich, R.; Smola, AJ; Helmbold, D.; Williamson, B., A generalized representer theorem, Computational Learning Theory, 416-426 (2001), Berlin Heidelberg: Springer, Berlin Heidelberg · Zbl 0992.68088 |
| [43] | Schölkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT press (2002) |
| [44] | Schölkopf, B.; Sung, KK; Burges, CJC; Girosi, F.; Niyogi, P.; Poggio, T.; Vapnik, V., Comparing support vector machines with Gaussian kernels to radial basis function classifiers, IEEE Transactions on Signal Processing, 45, 11, 2758-2765 (1997) |
| [45] | Schuster, T., Kaltenbacher, B., Hofmann, B., Kazimierski, K.S.: Regularization Methods in Banach Spaces, vol. 10. Walter de Gruyter (2012) · Zbl 1259.65087 |
| [46] | Schwartz, L., Théorie des Distributions (1966), Paris: Hermann, Paris · Zbl 0149.09501 |
| [47] | Singer, I., Best Approximation in Normed Linear Spaces by Elements of Linear Subspaces (1970), Berlin: Springer-Verlag, Berlin · Zbl 0197.38601 |
| [48] | Steinke, F.; Schölkopf, B., Kernels, regularization and differential equations, Pattern Recognition, 41, 11, 3271-32286 (2008) · Zbl 1154.68472 |
| [49] | Tibshirani, R., Regression shrinkage and selection via the Lasso, Journal of the Royal Statistical Society. Series B, 58, 1, 265-288 (1996) · Zbl 0850.62538 |
| [50] | Tikhonov, AN, Solution of incorrectly formulated problems and the regularization method, Soviet Mathematics, 4, 1035-1038 (1963) · Zbl 0141.11001 |
| [51] | Unser, M., A representer theorem for deep neural networks, Journal of Machine Learning Research, 20, 110, 1-30 (2019) · Zbl 1434.68526 |
| [52] | Unser, M.; Fageot, J.; Gupta, H., Representer theorems for sparsity-promoting \(\ell_1\) regularization, IEEE Transactions on Information Theory, 62, 9, 5167-5180 (2016) · Zbl 1359.94185 |
| [53] | Unser, M.; Fageot, J.; Ward, JP, Splines are universal solutions of linear inverse problems with generalized-TV regularization, SIAM Review, 59, 4, 769-793 (2017) · Zbl 1382.41011 |
| [54] | Wahba, G., Spline Models for Observational Data (1990), Philadelphia, PA: Society for Industrial and Applied Mathematics, Philadelphia, PA · Zbl 0813.62001 |
| [55] | Xu, Y., Ye, Q.: Generalized Mercer kernels and reproducing kernel Banach spaces, vol. 258. Memoirs of the American Mathematical Society (2019) · Zbl 1455.68009 |
| [56] | Zhang, H.; Xu, Y.; Zhang, J., Reproducing kernel Banach spaces for machine learning, Journal of Machine Learning Research, 10, 2741-2775 (2009) · Zbl 1235.68217 |
| [57] | Zhang, H.; Zhang, J., Regularized learning in Banach spaces as an optimization problem: representer theorems, Journal of Global Optimization, 54, 2, 235-250 (2012) · Zbl 1281.90088 |
| [58] | Zhang, H.; Zhang, J., Vector-valued reproducing kernel Banach spaces with applications to multi-task learning, Journal of Complexity, 29, 2, 195-215 (2013) · Zbl 1323.46030 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
