Approximation of optimal feedback control: a dynamic programming approach. (English) Zbl 1185.49030
Summary: We consider the general continuous time finite-dimensional deterministic system under a finite horizon cost functional. Our aim is to calculate approximate solutions to the optimal feedback control. First we apply the dynamic programming principle to obtain the evolutive Hamilton-Jacobi-Bellman (HJB) equation satisfied by the value function of the optimal control problem. We then propose two schemes to solve the equation numerically. One is the time difference approximation and the other the time-space approximation. For each scheme, we prove that (a) the algorithm is convergent, that is, the solution of the discrete scheme converges to the viscosity solution of the HJB equation, and (b) the optimal control of the discrete system determined by the corresponding dynamic programming is a minimizing sequence of the optimal feedback control of the continuous counterpart. An example is presented for the time-space algorithm; the results illustrate that the scheme is effective.
MSC:
| 49L20 | Dynamic programming in optimal control and differential games |
| 49L25 | Viscosity solutions to Hamilton-Jacobi equations in optimal control and differential games |
| 34H05 | Control problems involving ordinary differential equations |
| 49M30 | Other numerical methods in calculus of variations (MSC2010) |
| 65L12 | Finite difference and finite volume methods for ordinary differential equations |
Keywords:
viscosity solution; Hamilton-Jacobi-Bellman equation; finite difference; optimal feedback controlReferences:
| [1] | Bardi M.: Some applications of viscosity solutions to optimal control and differential games. In: Dolcetta, I.C., Lions, P.L. (eds) Viscosity Solutions and Applications, Lecture Notes in Mathematics, vol. 1660, pp. 44–97. Springer, Berlin (1997) · Zbl 0884.49020 |
| [2] | Bardi M., Dolcetta I.C.: Optimal Control and Viscosity Solutions of Hamilton–Jacobi–Bellman Equations. Birkhauser, Boston (1997) · Zbl 0890.49011 |
| [3] | Barles G., Souganidis P.E.: Convergence of approximation schemes for fully nonlinear second order equations. J. Asymptot. Anal. 4, 271–283 (1991) · Zbl 0729.65077 |
| [4] | Barron E.N.: Application of viscosity solutions of infinite-dimensional Hamilton–Jacobi–Bellman equations to some problems in distributed optimal control. J. Optim. Theory Appl. 64, 245–268 (1990) · Zbl 0687.49022 · doi:10.1007/BF00939448 |
| [5] | Bryson A.E. Jr: Optimal control–1950–1985. IEEE Control Syst. Mag. 13, 26–33 (1996) |
| [6] | Cannarsa P., Gozzi F., Soner H.M.: A dynamic programming approach to nonlinear boundary control problems of parabolic type. J. Funct. Anal. 117, 25–61 (1993) · Zbl 0823.49017 · doi:10.1006/jfan.1993.1122 |
| [7] | Capuzzo Dolcetta I.: On a descrete approximation of the Hamilton–Jacobi–Bellman equation of dynamic programming. Appl. Math. Optim. 10, 366–377 (1983) · Zbl 0582.49019 · doi:10.1007/BF01448394 |
| [8] | Capuzzo Dolcetta I., Ishii H.: Approximation solutions of the Bellman equation of deterministic control theory. Appl. Math. Optim. 11, 161–181 (1984) · Zbl 0553.49024 · doi:10.1007/BF01442176 |
| [9] | Crandall M.G.: Viscosity solutions: a primer. In: Dolcetta, I.C., Lions, P.L. (eds) Viscosity Solutions and Applications, Lecture Notes in Mathematics, vol. 1660, pp. 1–43. Springer, Berlin (1997) · Zbl 0901.49026 |
| [10] | Crandall M.G., Evans L.C., Lions P.L.: Some properties of viscosity solutions of Hamilton–Jacobi equations. Tran. Amer. Math. Soc. 282, 487–502 (1984) · Zbl 0543.35011 · doi:10.1090/S0002-9947-1984-0732102-X |
| [11] | Crandall M.G., Ishii H., Lions P.L.: User’s guide to viscosity solutions of second order partial differential equations. Bull. Amer. Math. Soc. 27, 1–67 (1992) · Zbl 0755.35015 · doi:10.1090/S0273-0979-1992-00266-5 |
| [12] | Crandall M.G., Lions P.L.: Viscosity solutions of Hamilton–Jacobi equations. Tran. Amer. Math. Soc. 277, 1–42 (1983) · Zbl 0599.35024 · doi:10.1090/S0002-9947-1983-0690039-8 |
| [13] | Crandall M.G., Lions P.L.: Two approximations of solutions of Hamilton–Jacobi equations. Math. Comp. 43, 1–19 (1984) · Zbl 0557.65066 · doi:10.1090/S0025-5718-1984-0744921-8 |
| [14] | Falcone, M.: A numerical approach to the infinite horizon problem of deterministic control theory. Appl. Math. Optim. 15, 1–13 (1987) and 23, 213–214 (1991) · Zbl 0715.49023 |
| [15] | Falcone, M.: Numerical solutions of dynamic programming equations, Appendix in the book by Bardi, M., Capuzzo Dolcetta, I. (eds.) Optimal Control and Viscosity Solutions of Hamilton–Jacobi–Bellman Equations. Birkhauser, Boston (1997) |
| [16] | Falcone, M., Giorgi, T.: An approximation scheme for evolutive Hamilton–Jacobi equations. In: Stochastic Analysis, Control, Optimization and applications, Systems and Control: Foundations and Applications, pp. 289–303. Birkhauser, Boston (1999) · Zbl 0931.65067 |
| [17] | Fleming W.H., Sonor H.M.: Controlled Markov Processes and Viscosity Solutions. 2nd edn. Springer, New York (2006) |
| [18] | Gozzi F., Sritharan S.S., Swiech A.: Viscosity solutions of dynamic-programming equations for the optimal control of the two-dimensional Navier–Stokes equations. Arch. Ration. Mech. Anal. 163, 295–327 (2002) · Zbl 1005.49028 · doi:10.1007/s002050200203 |
| [19] | Guo B.Z., Sun B.: Numerical solution to the optimal birth feedback control of a population dynamics: viscosity solution approach. Optim. Control Appl. Meth. 26, 229–254 (2005) · doi:10.1002/oca.759 |
| [20] | Guo B.Z., Sun B.: Numerical solution to the optimal feedback control of continuous casting process. J. Glob. Optim. 39, 171–195 (2007) · Zbl 1123.49024 · doi:10.1007/s10898-006-9130-0 |
| [21] | Guo, B.Z., Sun, B.: A new algorithm for finding numerical solutions of optimal feedback control. IMA Math.Control Inf. (to appear) · Zbl 1160.49037 |
| [22] | Huang C.S., Wang S., Teo K.L.: Solving Hamilton–Jacobi–Bellman equations by a modified method of characteristics. Nonlinear Anal., TMA 40, 279–293 (2000) · Zbl 0959.49021 · doi:10.1016/S0362-546X(00)85016-6 |
| [23] | Huang C.S., Wang S., Teo K.L.: On application of an alternating direction method to Hamilton–Jacobi–Bellman equations. J. Comput. Appl. Math. 166, 153–166 (2004) · Zbl 1044.65056 · doi:10.1016/j.cam.2003.09.031 |
| [24] | Kocan M., Soravia P.: A viscosity approach to infinite-dimensional Hamilton–Jacobi equations arising in optimal control with state constraints. SIAM J. Control Optim. 36, 1348–1375 (1998) · Zbl 0918.49025 · doi:10.1137/S0363012996301622 |
| [25] | Lions P.L.: Generalized Solutions of Hamilton–Jacobi Equations. Pitman, London (1982) · Zbl 0497.35001 |
| [26] | Loxton R.C., Teo K.L., Rehbock V.: Optimal control problems with multiple characteristic time points in the objective and constraints. Automatica 44, 2923–2929 (2008) · Zbl 1160.49033 · doi:10.1016/j.automatica.2008.04.011 |
| [27] | Rubio J.E.: Control and Optimization: The Linear Treatment of Nonlinear Problems, Nonlinear Science: Theory and Applications. Manchester University Press, Manchester (1986) |
| [28] | Souganidis P.E.: Approximation schemes for viscosity solutions of Hamilton–Jacobi equations. J. Differ. Equ. 59, 1–43 (1985) · doi:10.1016/0022-0396(85)90136-6 |
| [29] | Stoer J., Bulirsch R.: Introduction to numerical analysis. 2nd edn. Springer, New York (1993) · Zbl 0771.65002 |
| [30] | von Stryk O., Bulirsch R.: Direct and indirect methods for trajectory optimization. Annu. Oper. Res. 37, 357–373 (1992) · Zbl 0784.49023 · doi:10.1007/BF02071065 |
| [31] | Sussmann H.J., Willems J.C.: 300 years of optimal control: from the brachystochrone to the maximum principle. IEEE Controls Syst. Mag. 17, 32–44 (1997) · Zbl 1014.49001 · doi:10.1109/37.588098 |
| [32] | Teo K.L., Goh C.J., Wong K.H.: A Unified Computational Approach to Optimal Control Problems. Longman Scientific and Technical, England (1991) · Zbl 0747.49005 |
| [33] | Wang S., Gao F., Teo K.L.: An upwind finite-difference method for the approximation of viscosity solutions to Hamilton–Jacobi–Bellman equations. IMA J. Math. Control. Inf. 17, 167–178 (2000) · Zbl 0952.49025 · doi:10.1093/imamci/17.2.167 |
| [34] | Wang S., Jennings L.S., Teo K.L.: Numerical solution of Hamilton–Jacobi–Bellman equations by an upwind finite volume method. J. Glob. Optim. 27, 177–192 (2003) · Zbl 1047.49026 · doi:10.1023/A:1024980623095 |
| [35] | Willamson W.E.: Use of polynomial approximations to calculate suboptimal controls. AIAA J. 9, 2271–2273 (1971) · Zbl 0239.49019 · doi:10.2514/3.6499 |
| [36] | Yong J.M.: Dynamic Programming Principle and Hamilton–Jacobi–Bellman Equations (in Chinese). Shanghai Scientific and Technical Publishers, Shanghai (1992) |
| [37] | Yong J.M., Zhou X.Y.: Stochastic Controls: Hamiltonian Systems and HJB Equations. Applications of Mathematics, vol. 43. Springer, New York (1999) · Zbl 0943.93002 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
