While this paper may be regarded in a sense as a continuation of the program begun by Fisher on ”information”, the author’s view is that models should not (nay, cannot) be restricted to have a fixed number of parameters: Fisher’s information is thus inapplicable, and complexity and information must be considered directly in the observed data.
After noting three different notions of complexity, the author defines the stochastic complexity of a string of data, relative to a class of probabilistic models, as ”the fewest number of binary digits with which the data can be encoded by taking advantage of the selected models.” From the main result it appears that scepticism of Bayesian attempts to introduce improper priors is in order. The formalization of the concepts ”useful information” and ”prior information” in the data is also undertaken.