Universal coding, information, prediction, and estimation. (English) Zbl 0574.62003
In this paper the author generalizes Shannon’s information in that it is defined relative to a parametric class of distributions rather than relative to a single distribution. Moreover, this is done without a Bayesian prior in the parameter space, which for that matter is not restricted to vectors of the same dimension.
Shannon’s fundamental code length inequality is also generalized, and it provides a means to assess the goodness of models even when they have different numbers of parameters. This notion of information turns out to link together all the central notions in signal processing. Later research has strengthened the results and generalized their scope to include all of statistical inquiry: the author, ”Stochastic complexity and modeling”, Ann. Stat. to appear Sept. 1986.
Shannon’s fundamental code length inequality is also generalized, and it provides a means to assess the goodness of models even when they have different numbers of parameters. This notion of information turns out to link together all the central notions in signal processing. Later research has strengthened the results and generalized their scope to include all of statistical inquiry: the author, ”Stochastic complexity and modeling”, Ann. Stat. to appear Sept. 1986.
MSC:
62B10 | Statistical aspects of information-theoretic topics |
94A17 | Measures of information, entropy |
94A24 | Coding theorems (Shannon theory) |
62A01 | Foundations and philosophical topics in statistics |
62F10 | Point estimation |