Concentration estimates for learning with unbounded sampling. (English) Zbl 1283.68289
Kernel based least square regression learning with unbounded sampling processes is studied. By a moment hypothesis for unbounded sampling, an approximation assumption for the regression function, and capacity condition for hypothesis space, sharper learning rates are derived. In the error analysis, a probability inequality for unbounded random variables is introduced, and an iteration technique is used.
Reviewer: Hongwei Sun (Jinan)
MSC:
| 68T05 | Learning and adaptive systems in artificial intelligence |
| 62J02 | General nonlinear regression |
Keywords:
learning theory; least-square regression; regularization in reproducing kernel Hilbert spaces; empirical covering number; concentration estimatesReferences:
| [1] | Anthony, M., Bartlett, P.L.: Neural Network Learning: Theoretical Foundations. Cambridge University Press (1999) · Zbl 0968.68126 |
| [2] | Aronszajn, N.: Theory of reproducing kernels. Trans. Am. Math. Soc. 68, 337–404 (1950) · Zbl 0037.20701 · doi:10.1090/S0002-9947-1950-0051437-7 |
| [3] | Bartlett, P.L., Bousquet, O., Mendelson, S.: Local Rademacher complexities. Ann. Stat. 33, 1497–1537 (2005) · Zbl 1083.62034 · doi:10.1214/009053605000000282 |
| [4] | Bennett, G.: Probability inequalities for the sum of independent random variables. J. Am. Stat. Assoc. 57, 33–45 (1962) · Zbl 0104.11905 · doi:10.1080/01621459.1962.10482149 |
| [5] | Caponnetto, A., De Vito, E.: Optimal retes for regularized least-squares algorithm. Found. Comput. Math. 7, 331–368 (2007) · Zbl 1129.68058 · doi:10.1007/s10208-006-0196-8 |
| [6] | Chen, D.R., Wu, Q., Ying, Y., Zhou, D.X.: Support vector machine soft margin classifiers: error analysis. J. Mach. Learn. Res. 5, 1143–1175 (2004) · Zbl 1222.68167 |
| [7] | Cucker, F., Smale, S.: On the mathematical foundations of learning. Bull. Am. Math. Soc. 39, 1–49 (2001) · Zbl 0983.68162 · doi:10.1090/S0273-0979-01-00923-5 |
| [8] | Cucker, F., Smale, S.: Best choices for regularization parameters in learning theory: on the bias-variance problem. Found. Comput. Math. 2, 413–428 (2002) · Zbl 1057.68085 · doi:10.1007/s102080010030 |
| [9] | De Mol, C., De Vito, E., Rosasco, L.: Elastic-net regularization in learning theory. J. Complex. 25, 201–230 (2009) · Zbl 1319.62087 · doi:10.1016/j.jco.2009.01.002 |
| [10] | De Vito, E., Caponnetto, A., Rosasco, L.: Model selection for regularized least-squares algorithm in learning theory. Found. Comput. Math. 5, 59–85 (2005) · Zbl 1083.68106 · doi:10.1007/s10208-004-0134-1 |
| [11] | Koltchinskii, V., Panchenko, D.: Complexities of convex combinations and bounding the generalization error in classification. Ann. Stat. 33, 1455–1496 (2005) · Zbl 1080.62045 · doi:10.1214/009053605000000228 |
| [12] | Mendelson, S., Neeman, J.: Regularization in kernel learning. Ann. Stat. 38, 526–565 (2010) · Zbl 1191.68356 · doi:10.1214/09-AOS728 |
| [13] | Pan, Z.W., Xiao, Q.W.: Least-square regularized regression with non-iid sampling. J. Stat. Plan. Inference 139, 3579–3587 (2009) · Zbl 1176.68163 · doi:10.1016/j.jspi.2009.04.007 |
| [14] | Pontil, M.: A note on different covering numbers in learning theory. J. Complex. 19, 665–671 (2003) · Zbl 1057.68044 · doi:10.1016/S0885-064X(03)00033-5 |
| [15] | Smale, S., Zhou, D.X.: Learning theory estimates via integral operators and their approximations. Constr. Approx. 26, 153–172 (2007) · Zbl 1127.68088 · doi:10.1007/s00365-006-0659-y |
| [16] | Smale, S., Zhou, D.X.: Online learning with Markov sampling. Anal. Appl. 7, 87–113 (2009) · Zbl 1170.68022 · doi:10.1142/S0219530509001293 |
| [17] | Steinwart, I., Hush, D., Scovel, C.: Optimal rates for regularized least-squares regression. In: Dasgupta, S., Klivans, A. (eds.) Proceedings of the 22nd Annual Conference on Learning Theory, pp. 79–93 (2009) |
| [18] | Steinwart, I., Scovel, C.: Fast rates for support vector machines. Lect. Notes Comput. Sci. 3559, 279–294 (2005) · Zbl 1137.68564 · doi:10.1007/11503415_19 |
| [19] | Sun, H., Wu, Q.: A note on application of integral operator in learning theory. Appl. Comput. Harmon. Anal. 26, 416–421 (2009) · Zbl 1165.68059 · doi:10.1016/j.acha.2008.10.002 |
| [20] | Temlyakov, V.N.: Approximation in learning theory. Constr. Approx. 27, 33–74 (2008) · Zbl 05264756 · doi:10.1007/s00365-006-0655-2 |
| [21] | van der Vaart, A.W., Wellner, J.A.: Weak Convergence and Empirical Processes. Springer, New York (1996) · Zbl 0862.60002 |
| [22] | Wang, C., Zhou, D.X.: Optimal learning rates for least square regularized regression with unbounded sampling. J. Complex. 27, 55–67 (2011) · Zbl 1217.65024 · doi:10.1016/j.jco.2010.10.002 |
| [23] | Wu, Q., Ying, Y., Zhou, D.X.: Learning rates of least-square regularized regression. Found. Comput. Math. 6, 171–192 (2006) · Zbl 1100.68100 · doi:10.1007/s10208-004-0155-9 |
| [24] | Wu, Q., Ying, Y., Zhou, D.X.: Multi-kernel regularized classifiers. J. Complex. 23, 108–134 (2007) · Zbl 1171.65043 · doi:10.1016/j.jco.2006.06.007 |
| [25] | Wu, Q., Zhou, D.X.: Learning with sample dependent hypothesis spaces. Comput. Math. Appl. 56, 2896–2907 (2008) · Zbl 1165.68388 · doi:10.1016/j.camwa.2008.09.014 |
| [26] | Wu, Z.M.: Compactly supported positive definite radial functions. Adv. Comput. Math. 4, 283–292 (1995) · Zbl 0837.41016 · doi:10.1007/BF03177517 |
| [27] | Xu, Y.L., Chen, D.R.: Learning rates of regularized regression for exponentially strongly mixing sequence. J. Stat. Plan. Inference 138, 2180–2189 (2008) · Zbl 1134.62050 · doi:10.1016/j.jspi.2007.09.003 |
| [28] | Zhang, T.: Leave-one-out bounds for kernel methods. Neural Comput. 15, 1397–1437 (2003) · Zbl 1085.68144 · doi:10.1162/089976603321780326 |
| [29] | Zhou, D.X.: Capacity of reproducing kernel spaces in learning theory. IEEE Trans. Inf. Theory 49, 1743–1752 (2003) · Zbl 1290.62033 · doi:10.1109/TIT.2003.813564 |
| [30] | Zhou, D.X.: Derivative reproducing properties for kernel methods in learning theory. J. Comput. Appl. Math. 220, 456–463 (2008) · Zbl 1152.68049 · doi:10.1016/j.cam.2007.08.023 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
