This is a reprint of English translation of Russian Edition 1977. The main problem addressed in the monograph is to “compute” the minimal value
\[ v(0,x):=\inf E \int_0^T f(\alpha_s, X^{\alpha}_s)\,ds, \tag{1} \]
where \(f\) is some “loss function”, and the process \(X^{\alpha}\) is a solution of Itô’s Stochastic Differential Equation (SDE) in \(\mathbb R^d\),
\[ dX^\alpha_s= \sigma(\alpha_s, X^\alpha_s)\,dW_s + b(\alpha_s, X^\alpha_s)\,ds,\quad s\geq 0, \qquad X^\alpha_0=x, \tag{2} \]
where \(W\) is a Wiener process, \(b\) is a vector-valued drift, and \(\sigma\) is a diffusion matrix, all of corresponding dimensions. What makes the equation and solution controlled is a strategy \(\alpha = (\alpha_s\), \(0\leq s\leq T)\) from some special class of processes, which is used for “tuning” both \(X^\alpha\) and \(f\), so as to minimize the expectation in the right hand side of (1) over this specified class. In the whole monograph it is assumed that, in particular, any strategy takes values from some given set \(A\). Here “compute” is understood in the sense of establishing a certain differential equation for this function \(v\) that could be solved, at least, theoretically (cf. also the remark in the end of this review). The function defined in (1) is just one example among other various versions of several similar functionals also considered in the book, such as with infinite horizon (\(T=\infty\)) and “discount”, with stopping times, with non-homogeneous coefficients, with “terminal payment”, etc. They are all called payoff functions or cost functions. Important is that the value \(v(0,x)\) can be effectively computed only if instead of one value we allow variable \(t\) and \(x\), that is, if we consider the whole function \((v(t,x)\), \(t\in [0,T]\), \(x\in\mathbb R^d)\). (Of course, for \(t>0\) the integral in (1) should be replaced by \(\int_t^T\), and the equation (2) should be started at time \(t\) instead of zero.) This is a deep analogue of (backward) Kolmogorov’s equation for Markov diffusions without control. In this way, the function \(v\) turns out to be a solution of some nonlinear Partial Differential Equation (PDE) called Bellman’s equation, in appropriate Sobolev function classes,
\[ \inf_{a\in A}(v_t(t,x)+ L^a v(t,x) + f(a,x))=0, \quad 0\leq t\leq T, \;x\in\mathbb R^d, \qquad v(T,x) = 0, \]
where
\[ L^a := \frac12 \, (\sigma\sigma^*(a,x))^{ij}\frac{\partial^2}{\partial x^i \partial x^j} + b^i(a,x)\frac{\partial}{\partial x^i }. \]
Hence, a possible point of view could be that the book treats a large class of fully nonlinear parabolic PDEs via probabilistic methods. Nonlinear elliptic problems are also considered. Here “fully nonlinear” is due to dependence of both coefficients \(\sigma\) and \(b\) on the strategy.
Chapter 1 – Introduction to the theory of controlled diffusion processes. This chapter can be recommended as a first reading for a PhD student working in the area. Main topics are Bellman’s principle sometimes called integral Bellman’s equation, Bellman’s differential equation, normed Bellman’s equation, applications to certain inequalities, one-dimensional theory, optimal stopping for 1D diffusion processes. Some parts of this chapter are written deliberately in a slightly non-rigorous style, so as to present involved ideas in a quick and simple fashion. Bellman’s differential equation is deduced from the Bellman principle via a limiting procedure studied in chapters 3 and 4. In any control theory an important player is strategy. A useful technical feature is that several different classes of strategies are studied, among those Markov strategies and some wider classes of strategies. Eventually in further chapters it will be proved that the payoff functions corresponding to all those different classes of strategies coincide; in other words, Markov strategies are sufficient (or efficient) in a much wider class of strategies which may depend on past, etc. This result has important implications in the theory of partial differential equations.
The remaining chapters 2–6 contain a rigorous presentation outlined in the first chapter and some extensions.
Chapter 2, Auxiliary propositions, is a collection of several fundamental results. First of all, these are versions of Krylov’s estimates of distributions of stochastic integrals. The base of all those estimates is the author’s earlier result on a parabolic version of the Alexandrov-Bakelman-Pucci inequalities for elliptic PDEs given here without proof as a Lemma. Notice some further use of Krylov’s estimates in the theory of PDEs where it leaded to Krylov-Safonov’s Harnack inequalities – not exploited in this book – for parabolic and elliptic PDEs for non-divergent differential operators of the second order. Also, the chapter provides the author’s result about existence of weak solution of an SDE with measurable coefficients and some other preparations to the next chapters, including Itô-Krylov’s formula for functions with Sobolev derivatives. The importance of the latter follows from the fact that even without control, the payoff function (function \(v\)) usually admits only second order Sobolev derivatives, but not classical ones. SDEs with measurable coefficients are very helpful because they allow any Markov control.
Chapter 3, General Properties of a Payoff Functions, and Chapter 4, The Bellman equation, contain step by step derivation of Bellman’s differential equation for the payoff function.
Chapter 5 suggests methods of construction of \(\varepsilon\)-optimal strategies which are Markov, based on Bellman’s equation. Finally, Chapter 6 suggests some improvements to Bellman’s equation for unbounded coefficients where the original version of this equation often fails.
There are two Appendices with standard facts about semimartingales and stochastic integrals. However, notice one simple and useful but “less known” feature of Itô’s stochastic integral, the formula (4) of Appendix 1. It states that “usual” stochastic integral, although it cannot be defined on trajectories, nevertheless may be regarded as a limit in probability of some specially designed Riemann type integral sums.
Among more recent developments related directly to the main topic of the book, it is worth mentioning the author’s series of works on finite difference approximations of solutions of Bellman’s equations [see, e.g., Probab. Theory Relat. Fields 117, No. 1, 1–16 (2000; Zbl 0971.65081)].
About a PDE approach to nonlinear Bellman’s equations see, in particular [Nonlinear elliptic and parabolic equations of the second order. Mathematics and its Applications (Soviet Series), Dordrecht etc.: D. Reidel Publishing Company (1987; Zbl 0619.35004)].
The decision of Springer to reprint this classical book is really timely. The monograph may be strongly recommended as an excellent reading to PhD students, postdocs et al. working in the area of controlled stochastic processes and/or nonlinear partial differential equations of the second order. The reader, of course, should be prepared to a hard work, which, however, could be highly rewarding. The first chapter may be recommended to a wider audience of all students specializing in stochastic analysis or stochastic finance starting from MSc level.