In this paper the author generalizes Shannon’s information in that it is defined relative to a parametric class of distributions rather than relative to a single distribution. Moreover, this is done without a Bayesian prior in the parameter space, which for that matter is not restricted to vectors of the same dimension.
Shannon’s fundamental code length inequality is also generalized, and it provides a means to assess the goodness of models even when they have different numbers of parameters. This notion of information turns out to link together all the central notions in signal processing. Later research has strengthened the results and generalized their scope to include all of statistical inquiry: the author, ”Stochastic complexity and modeling”, Ann. Stat. to appear Sept. 1986.