Let \(\mathfrak E\) be the real axis and \(\mathfrak B\) a certain \(\sigma\)-algebra of its subsets, \(\xi\) and \(\eta\) two real-valued random variables, \(P_\xi\), \(P_\eta\) the probability distributions of \(\xi\) and \(\eta\) and \(P_{\xi\eta}\) the joint probability distribution function of \(\xi\) and \(\eta\). Denote by
\[ J_1(\xi,\eta)=\sup\sum_{i=1}^n P_{\xi\eta}(S_i) \log \frac{P_{\xi\eta}(S_i)}{P_\xi P_\eta(S_i)} \]
the quantity of information contained in \(\xi\) with respect to \(\eta\), where \(\sup\) is taken over all possible partitions of \(\mathfrak E\times\mathfrak E\) into a finite number of mutually disjoint sets \(S_i\) \((1\leq i\leq n)\) of the \(\sigma\)-algebra \(\mathfrak B\times\mathfrak B\), and by \(J_2(\xi,\eta)\) and \(J_3(\xi,\eta)\) the corresponding quantities for which the above mentioned \(\sup\) is taken, respectively, over all generalized rectangles \((S_i = B\times B'\), \(B, B'\in\mathfrak B)\) and over all rectangles \((S_i =\Delta\times\Delta'\), where \(\Delta\) and \(\Delta'\) are intervals). If \(P_{\xi\eta}\) is absolutely continuous with respect to \(P_\xi P_\eta\), then
\[ J_2(\xi,\eta)= {\int\int}_{\mathfrak E\times\mathfrak E} \alpha(x,y) \log\alpha(x,y) P_\xi(dx) P_\eta(dy),\quad \alpha(x,y)=\frac {d^2P_{\xi\eta}}{dP_\xi dP_\eta} \]
and \(J_1(\xi,\eta)= J_2(\xi,\eta)\). Suppose now that \(P_{\xi\eta}\) is absolutely continuous with respect to \(P_\xi\), \(P_\eta\) and \(\mathfrak B\) is the \(\sigma\)-algebra generated by all the intervals; the conditions here imposed imply
\[ J_3(\xi,\eta)={\int\int}_{\mathfrak E\times\mathfrak E} \alpha(x,y) \log\alpha(x,y) P_\xi(dx) P_\eta(dy) \]
and hence \(J_1(\xi,\eta)= J_2(\xi,\eta)= J_3(\xi,\eta)\). If \(P_{\xi\eta}\) is not absolutely continuous with respect to \(P_\xi P_\eta\), it follows that \(J_1(\xi,\eta)= J_2(\xi,\eta)= J_3(\xi,\eta)=\infty\). (See also I. M. Gel’fand and A. M. Yaglom [Usp. Mat. Nauk 12, No. 1(73), 3–52 (1957; Zbl 0078.32203)].