From the paper: In the field of neural networks, so-called infomax principles like the principle of “maximum information preservation” by R. Linsker [Computer 21, 105-117 (1988)] are formulated to derive learning rules that improve the information processing properties of neural systems. These principles, which are based on information-theoretic measures, are intended to describe the mechanism of learning in the brain. There, the starting point is a low-dimensional and biophysiologically motivated parametrization of the neural system, which need not necessarily be compatible with the given optimization principle. In contrast to this, we establish theoretical results about the low complexity of optimal solutions for the optimization problem of frequently used measures like the mutual information in an unconstrained and more theoretical setting. We do not comment on applications to modeling neural networks.
Within the framework of information geometry, the interaction among units of a stochastic system is quantified in terms of the Kullback-Leibler divergence of the underlying joint probability distribution from an appropriate exponential family. In the present paper, the main example for such a family is given by the set of all factorizable random fields. Motivated by this example, the locally farthest points from an arbitrary exponential family \({\mathcal E}\) are studied. In the corresponding dynamical setting, such points can be generated by the structuring process with respect to \({\mathcal E}\) as a repelling set. The main results concern the low complexity of such distributions which can be controlled by the dimension of \({\mathcal E}\).