Interpreting Kullback–Leibler divergence with the Neyman-Pearson Lemma. (English) Zbl 1101.62004
Summary: Kullback-Leibler divergence and the Neyman-Pearson lemma are two fundamental concepts in statistics. Both are about likelihood ratios: Kullback-Leibler divergence is the expected log-likelihood ratio, and the Neyman-Pearson lemma is about error rates of likelihood ratio tests. Exploring this connection gives another statistical interpretation of the Kullback-Leibler divergence in terms of the loss of power of the likelihood ratio test when the wrong distribution is used for one of the hypotheses. In this interpretation, the standard non-negativity property of the Kullback-Leibler divergence is essentially a restatement of the optimal property of likelihood ratios established by the Neyman-Pearson lemma. The asymmetry of Kullback-Leibler divergence is overviewed in information geometry.
Keywords:
exponential connection; mixture connection; information geometry; testing hypotheses; maximum likelihood; ROC curve; discriminant analysisReferences:
| [1] | Akaike, H., Information theory and an extension of the maximum likelihood principle, (Petrov, B. N.; Csáki, F., Proceedings of Second International Symposium on Inference Theory (1973), Académiai Kiadó: Académiai Kiadó Budapest), 67-281 · Zbl 0283.62006 |
| [2] | Amari, S., Differential-Geometrical Methods in Statistics, Lecture Notes in Statistics, vol. 28 (1985), Springer: Springer New York · Zbl 0559.62001 |
| [3] | Amari, S.; Nagaoka, H., Methods of Information Geometry, Translations of Mathematical Monographs, vol. 191 (2000), Oxford University Press: Oxford University Press Oxford · Zbl 0960.62005 |
| [4] | Cox, D. R.; Hinkley, D. V., Theoretical Statistics (1974), Chapman & Hall: Chapman & Hall London · Zbl 0334.62003 |
| [5] | Eguchi, S., Second order efficiency of minimum contrast estimators in a curved exponential family, Ann. Statist., 11, 793-803 (1983) · Zbl 0519.62027 |
| [6] | Eguchi, S., Geometry of minimum contrast, Hiroshima Math. J., 22, 631-647 (1992) · Zbl 0780.53015 |
| [7] | S. Eguchi, Information Geometry and Statistical Pattern Recognition, Sugaku Exposition, AMS, Providence, RI, 2005, to appear.; S. Eguchi, Information Geometry and Statistical Pattern Recognition, Sugaku Exposition, AMS, Providence, RI, 2005, to appear. · Zbl 1277.62164 |
| [8] | Eguchi, S.; Copas, J. B., Recent developments in discriminant analysis from an information geometric point of view, J. Korean Statist. Soc., 30, 247-264 (2001) |
| [9] | Eguchi, S.; Copas, J. B., A class of logistic type discriminant functions, Biometrika, 89, 1-22 (2002) · Zbl 0995.62065 |
| [10] | Kullback, S.; Leibler, R. A., On information and sufficiency, Ann. Math. Statist., 55, 79-86 (1951) · Zbl 0042.38403 |
| [11] | Lindsey, J. K., Parametric Statistical Inference (1996), Oxford University Press: Oxford University Press Oxford · Zbl 0855.62002 |
| [12] | Murata, N.; Takenouchi, T.; Kanamori, T.; Eguchi, S., Information geometry of U-Boost and Bregman divergence, Neural Comput., 16, 1437-1481 (2004) · Zbl 1102.68489 |
| [13] | Rao, C. R., Linear Statistical Inference and its Applications (2002), Wiley-Interscience: Wiley-Interscience New York · Zbl 0169.21302 |
| [14] | Wald, A., Note on the consistency of the maximum likelihood estimate, Ann. Math. Statist., 20, 595-601 (1949) · Zbl 0034.22902 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
