A novel SVM+NDA model for classification with an application to face recognition. (English) Zbl 1225.68230
Summary: Support vector machine (SVM) is a powerful classification methodology, where the support vectors fully describe the decision surface by incorporating local information. On the other hand, nonparametric discriminant analysis (NDA) is an improvement over LDA where the normality assumption is relaxed. NDA also detects the dominant normal directions to the decision plane. This paper introduces a novel SVM+NDA model which can be viewed as an extension to the SVM by incorporating some partially global information, especially, discriminatory information in the normal direction to the decision boundary. This can also be considered as an extension to the NDA where the support vectors improve the choice of \(k\)-nearest neighbors on the decision boundary by incorporating local information. Being an extension to both SVM and NDA, it can deal with heteroscedastic and non-normal data. It also avoids the small sample size problem. Moreover, it can be reduced to the classical SVM model, so that existing softwares can be used. A kernel extension of the model, called KSVM+KNDA is also proposed to deal with nonlinear problems. We have carried an extensive comparison of the SVM+NDA to the LDA, SVM, heteroscedastic LDA (HLDA), NDA and the combined SVM and LDA on artificial, real and face recognition data sets. Results for KSVM+KNDA have also been presented. These comparisons demonstrate the advantages and superiority of our proposed model.
MSC:
| 68T10 | Pattern recognition, speech recognition |
| 68T05 | Learning and adaptive systems in artificial intelligence |
Keywords:
linear discriminant analysis (LDA); nonparametric discriminant analysis (NDA); support vector machines (SVM); kernel NDA and SVM; small sample size problem; face recognitionReferences:
| [1] | A. Asuncion, D.J. Newman, UCI machine learning repository, University of California, Irvine, School of Information and Computer Sciences, 2007.; A. Asuncion, D.J. Newman, UCI machine learning repository, University of California, Irvine, School of Information and Computer Sciences, 2007. |
| [2] | A. Jain, R. Bolle, S. Pankanti (Eds.), BIOMETRIC—Personal Identification in Networked Society, Kluwer Academic Publishers, London, UK, 1999.; A. Jain, R. Bolle, S. Pankanti (Eds.), BIOMETRIC—Personal Identification in Networked Society, Kluwer Academic Publishers, London, UK, 1999. |
| [3] | A.S. Georghiades, Yale face database, Yale University, Center for Computational Vision and Control, 1997.; A.S. Georghiades, Yale face database, Yale University, Center for Computational Vision and Control, 1997. |
| [4] | Belegundu, A. D.; Chandrupatla, T. R., Optimization Concepts and Applications in Engineering (1999), Prentice-Hall · Zbl 0941.90074 |
| [5] | Lee, C.; Landgrebe, D. A., Feature extraction based on decision boundaries, IEEE Transactions on Pattern Analysis and Machine Intelligence, 15, 4, 388-400 (1993) |
| [6] | Park, C. H.; Park, H., Nonlinear discriminant analysis using kernel functions and the generalized singular value decomposition, SIAM Journal on Matrix Analysis and Application, 27, 1, 98-102 (2005) · Zbl 1095.65037 |
| [7] | Baudat, G.; Anouar, B., Generalized discriminant analysis using kernel approach, Neural Computation, 2385-2404 (2000) |
| [8] | Ratsch, G.; Onoda, T.; Muller, K. R., Soft margins for adaboost, Machine Learning, 42, 3, 287-320 (2000) · Zbl 0969.68128 |
| [9] | Golub, G. H.; Van Loan, C. F., Matrix Computations (1996), The John Hopkins University Press · Zbl 0865.65009 |
| [10] | Yu, H.; Yang, J., A direct LDA algorithm for high-dimensional data with application to face recognition, Pattern Recognition, 34, 2067-2070 (2001) · Zbl 0993.68091 |
| [11] | Kuhn, H. W.; Tucker, A. W., Nonlinear programming, (Proceedings of the Second Berkeley Symposium (1950)), 481-492 · Zbl 0044.05903 |
| [12] | Xu, J.; Zhang, X.; Li, Y., Kernel MSE algorithm: a unified framework for KFD, LS-SVM and KRR, (Proceedings of the International Joint Conference on Neural Networks, vol. 2 (2001)), 1486-1491 |
| [13] | Suykens, J. A.K.; Gestel, T. V.; Brabanter, J. D.; Moor, B. D.; Vandewalle, J., Least Squares Support Vector Machines (2002), World Scientific Pub. Co. · Zbl 1017.93004 |
| [14] | J.J. Weng, Crescepton and SHOSLIF: towards comprehensive visual learning, in: Early Visual Learning, 1996, pp. 183-214.; J.J. Weng, Crescepton and SHOSLIF: towards comprehensive visual learning, in: Early Visual Learning, 1996, pp. 183-214. |
| [15] | Fukunaga, K., Introduction to Statistical Pattern Recognition (2000), Academic Press |
| [16] | Fukunaga, K.; Hostetler, L., The estimation of the gradient of a density function, with applications in pattern recognition, IEEE Transactions on Information Theory, 21, 1, 32-40 (1975) · Zbl 0297.62025 |
| [17] | L. Spacek, Face recognition data, University of Essex, Computer Vision Science Research Projects, 2008.; L. Spacek, Face recognition data, University of Essex, Computer Vision Science Research Projects, 2008. |
| [18] | Cristianini, M.; Shawe-Taylor, J., An Introduction to Support Vector Machines (2000), Cambridge University Press: Cambridge University Press Cambridge, UK |
| [19] | Loog, M.; Duin, R. P.W., Linear dimensionality reduction via heteroscedastic extension of LDA: the Chernoff criterion, IEEE Transactions on Pattern Analysis and Machine Intelligence, 26, 6, 732-739 (2004) |
| [20] | Lyons, M.; Akamatsu, S.; Kamachi, M.; Gyoba, J., Coding facial expressions with Gabor wavelets, (Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition (1998)), 200-205 |
| [21] | MATLAB Bioinformatics \(\operatorname{Toolbox}^™ \operatorname{MathWorks}^™\); MATLAB Bioinformatics \(\operatorname{Toolbox}^™ \operatorname{MathWorks}^™\) |
| [22] | Lyons, M. J.; Budynek, J.; Akamatsu, S., Automatic classification of single facial images, IEEE Transactions on Pattern Analysis and Machine Intelligence, 21, 12, 1357-1362 (1999) |
| [23] | Yiu, M. L.; Mamoulis, N., Iterative projected clustering by subspace mining, IEEE Transactions on Knowledge and Data Engineering, 17, 2, 176-189 (2005) |
| [24] | Belhumeur, P. N.; Hespanha, J. P.; Kriegman, D. J., Eigenfaces vs. Fisherfaces: recognition using class specific linear projection, IEEE Transactions on Pattern Analysis and Machine Intelligence, 19, 711-720 (1997) |
| [25] | Duda, R. O.; Hart, P. E.; Stork, D. G., Pattern Classification (2000), Wiley-Interscience |
| [26] | Mika, S.; Ratsch, G.; Weston, J.; Scholkopf, B.; Mullers, K. R., Fisher discriminant analysis with kernels, (Proceedings of IEEE Neural Networks for Signal Processing Workshop (1999)), 41-48 |
| [27] | Shivaswamy, P. K.; Jebara, T., Maximum relative margin and data-dependent regularization, Journal of Machine Learning Research, 11, 747-788 (2010) · Zbl 1242.68249 |
| [28] | Xiong, T.; Ye, J.; Li, Q.; Cherkassky, V.; Janardan, R., Efficient kernel discriminant analysis via QR decomposition, (Advances in Neural Information Processing Systems, vol. 17 (2005)), 1529-1536 |
| [29] | Xiong, T.; Cherkassky, V., A combined SVM and LDA approach for classification, (Proceedings of the International Joint Conference on Neural Networks (2005)), 1455-1459 |
| [30] | Coleman, T. F.; Li, Y., A reflective Newton method for minimizing a quadratic function subject to bounds on some of the variables, SIAM Journal on Optimization, 6, 4, 1040-1058 (1996) · Zbl 0861.65053 |
| [31] | Lim, T. S.; Loh, W. Y.; Shih, Y. S., A comparison of prediction accuracy, complexity, and training time of thirty-three old and new classification algorithms, Machine Learning, 40, 3, 203-228 (2000) · Zbl 0969.68669 |
| [32] | Vapnik, V. N., Statistical Learning Theory (1998), John Wiley & Sons: John Wiley & Sons New York, USA · Zbl 0935.62007 |
| [33] | Press, W. H.; Teukolsky, S. A.; Vetterling, W. T.; Flannery, B. P., Numerical Recipes: The Art of Scientific Computing (2007), Cambridge University Press · Zbl 1132.65001 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
