Convergence of a robust deep FBSDE method for stochastic control. (English) Zbl 1511.93120
Summary: In this paper, we propose a deep learning based numerical scheme for strongly coupled forward backward stochastic differential equations (FBSDEs), stemming from stochastic control. It is a modification of the deep BSDE method in which the initial value to the backward equation is not a free parameter, and with a new loss function being the weighted sum of the cost of the control problem, and a variance term which coincides with the mean squared error in the terminal condition. We show by a numerical example that a direct extension of the classical deep BSDE method to FBSDEs fails for a simple linear-quadratic control problem, and we motivate why the new method works. Under regularity and boundedness assumptions on the exact controls of time continuous and time discrete control problems, we provide an error analysis for our method. We show empirically that the method converges for three different problems, one being the one that failed for a direct extension of the deep BSDE method.
MSC:
| 93E03 | Stochastic systems in control theory (general) |
| 60H30 | Applications of stochastic analysis (to PDEs, etc.) |
| 68T07 | Artificial neural networks and deep learning |
| 60H35 | Computational methods for stochastic equations (aspects of stochastic analysis) |
References:
| [1] | Andersson, K., Approximate Stochastic Control Based on Deep Learning and Forward Backward Stochastic Differential Equations, Master’s thesis, Chalmers University of Technology, 2019. |
| [2] | Antonelli, F., Backward Forward Stochastic Differential Equations, Ph.D. thesis, Purdue University, 1993. · Zbl 0780.60058 |
| [3] | Åström, K. J., Introduction to Stochastic Control Theory, Courier Corporation, 2012. |
| [4] | Beck, C., Becker, S., Cheridito, P., Jentzen, A., and Neufeld, A., Deep splitting method for parabolic PDEs, SIAM J. Sci. Comput., 43 (2021), pp. A3135-A3154. · Zbl 1501.65054 |
| [5] | Beck, C., Becker, S., Grohs, P., Jaafari, N., and Jentzen, A., Solving the Kolmogorov PDE by means of deep learning, J. Sci. Comput., 88 (2021), pp. 1-28. · Zbl 1490.65006 |
| [6] | Beck, C., Hutzenthaler, M., Jentzen, A., and Kuckuck, B., An Overview on Deep Learning-Based Approximation Methods for Partial Differential Equations, preprint, https://arxiv.org/abs/2012.12348, 2020. |
| [7] | Beck, C., Weinan, E., and Jentzen, A., Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations, J. Nonlinear. Sci., 29 (2019), pp. 1563-1619. · Zbl 1442.91116 |
| [8] | Bender, C. and Denk, R., A forward scheme for backward SDEs, Stochastic Process. Appl., 117 (2007), pp. 1793-1812. · Zbl 1131.60054 |
| [9] | Bender, C. and Steiner, J., Least-squares Monte Carlo for backward SDE, in Numerical Methods in Finance, Springer, New York, 2012, pp. 257-289. · Zbl 1269.91098 |
| [10] | Berner, J., Grohs, P. and Jentzen, A., Analysis of the generalization error: Empirical risk minimization over deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations, SIAM J. Math. Data Sci., 2 (2020), pp. 631-657. · Zbl 1480.60191 |
| [11] | Beyn, W.-J. and Kruse, R., Two-sided error estimates for the stochastic theta method, Discrete Contin. Dyn. Syst. Ser. B, 14 (2010), pp. 389-407. · Zbl 1195.65006 |
| [12] | Bonnans, J. F., Gianatti, J., and Silva, F. J., On the time discretization of stochastic optimal control problems: The dynamic programming approach, ESAIM Control Optim. Calc. Var., 25 (2019), 63. · Zbl 1447.93373 |
| [13] | Bouchard, B. and Touzi, N., Discrete-time approximation and Monte-Carlo simulation of backward stochastic differential equations, Stochastic Process. Appl., 111 (2004), pp. 175-206. · Zbl 1071.60059 |
| [14] | Buttazzo, G. and Maso, G. Dal, \( \gamma \) -convergence and optimal control problems, J. Optim. Theory Appl., 38 (1982), pp. 385-407. · Zbl 0471.49012 |
| [15] | Chan-Wai-Nam, Q., Mikael, J., and Warin, X., Machine learning for semi linear PDEs, J. Sci. Comput., 79 (2019), pp. 1667-1712. · Zbl 1433.68332 |
| [16] | Chau, K. W. and Oosterlee, C. W., Stochastic grid bundling method for backward stochastic differential equations, Int. J. Comput. Math., 96 (2019), pp. 2272-2301. · Zbl 1499.91165 |
| [17] | Chessari, J. and Kawai, R., Numerical Methods for Backward Stochastic Differential Equations: A Survey, preprint, https://arxiv.org/abs/2101.08936, 2021. |
| [18] | Crisan, D., Manolarakis, K., and Touzi, N., On the Monte Carlo simulation of BSDEs: An improvement on the Malliavin weights, Stochastic Process. Appl., 120 (2010), pp. 1133-1158. · Zbl 1193.65005 |
| [19] | Cybenko, G., Approximations by superpositions of a sigmoidal function, Math. Control Signals Systems, 2 (1989), pp. 183-192. |
| [20] | Dai, B., Surabhi, V. R., Krishnamurthy, P., and Khorrami, F., Learning Locomotion Controllers for Walking Using Deep FBSDE, preprint, https://arxiv.org/abs/2107.07931, 2021. |
| [21] | Elbrächter, D., Grohs, P., Jentzen, A., and Schwab, C., DNN expression rate analysis of high-dimensional PDEs: Application to option pricing, Constr. Approx., 55 (2022), pp. 3-71. · Zbl 1500.35009 |
| [22] | Fahim, A., Touzi, N., and Warin, X., A probabilistic numerical method for fully nonlinear parabolic PDEs, Ann. Appl. Probab., 21 (2011), pp. 1322-1364. · Zbl 1230.65009 |
| [23] | Fang, F. and Oosterlee, C. W., A novel pricing method for European options based on Fourier-cosine series expansions, SIAM J. Sci. Comput., 31 (2009), pp. 826-848. · Zbl 1186.91214 |
| [24] | Fujii, M., Takahashi, A., and Takahashi, M., Asymptotic expansion as prior knowledge in deep learning method for high dimensional BSDEs, Asia-Pac. Financ. Mark., 26 (2019), pp. 391-408. · Zbl 1422.91694 |
| [25] | Gnoatto, A., Reisinger, C., and Picarelli, A., Deep xVA Solver—A Neural Network Based Counterparty Credit Risk Management Framework, SSRN 3594076, 2020. |
| [26] | Gobet, E. and Labart, C., Error expansion for the discretization of backward stochastic differential equations, Stochastic Process. Appl., 117 (2007), pp. 803-829. · Zbl 1117.60058 |
| [27] | Gobet, E. and Labart, C., Solving BSDE with adaptive control variate, SIAM J. Numer. Anal., 48 (2010), pp. 257-277. · Zbl 1208.60055 |
| [28] | Gobet, E., López-Salas, J. G., Turkedjiev, P., and Vázquez, C., Stratified regression Monte Carlo scheme for semilinear PDEs and BSDEs with large scale parallelization on GPUs, SIAM J. Sci. Comput., 38 (2016), pp. C652-C677. · Zbl 1352.65008 |
| [29] | Grohs, P., Hornung, F., Jentzen, A., and von Wurstemberger, P., A Proof that Artificial Neural Networks Overcome the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equations, https://arxiv.org/abs/1809.02362, 2018. |
| [30] | Han, J., Deep Learning Approximation for Stochastic Control Problems, preprint, https://arxiv.org/abs/1611.07422, 2016. |
| [31] | Han, J., Jentzen, A., and Weinan, E., Solving high-dimensional partial differential equations using deep learning, Proc. Natl. Acad. Sci. USA, 115 (2018), pp. 8505-8510. · Zbl 1416.35137 |
| [32] | Han, J. and Long, J., Convergence of the deep BSDE method for coupled FBSDEs, Probab. Uncertain. Quant. Risk, 5 (2020), pp. 1-33. · Zbl 1454.60105 |
| [33] | Henry-Labordere, P., Deep Primal-Dual Algorithm for BSDEs: Applications of Machine Learning to CVA and IM, SSRN 3071506, 2017. |
| [34] | Huijskens, T., Ruijter, M. J., and Oosterlee, C. W., Efficient numerical fourier methods for coupled forward-backward SDEs, J. Comput. Appl. Math., 296 (2016), pp. 593-612. · Zbl 1336.65010 |
| [35] | Huré, C., Pham, H., and Warin, X., Some Machine Learning Schemes for High-Dimensional Nonlinear PDEs, preprint, https://arxiv.org/abs/1902.01599, 2019. |
| [36] | Huré, C., Pham, H., and Warin, X., Deep backward schemes for high-dimensional nonlinear PDEs, Math. Comp., 89 (2020), pp. 1547-1579. · Zbl 1440.60063 |
| [37] | Hutzenthaler, M., Jentzen, A., and Kruse, T., On multilevel Picard numerical approximations for high-dimensional nonlinear parabolic partial differential equations and high-dimensional nonlinear backward stochastic differential equations, J. Sci. Comput., 79 (2019), pp. 1534-1571. · Zbl 1418.65149 |
| [38] | Hutzenthaler, M., Jentzen, A., Kruse, T., and Nguyen, T. A., A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations, SN Partial Differ. Equ. Appl., 1 (2020), pp. 1-34. · Zbl 1455.65200 |
| [39] | Jentzen, A., Salimova, D., and Welti, T., A Proof that Deep Artificial Neural Networks Overcome the Curse of Dimensionality in the Numerical Approximation of Kolmogorov Partial Differential Equations with Constant Diffusion and Nonlinear Drift Coefficients, preprint, https://arxiv.org/abs/1809.07321, 2018. |
| [40] | Ji, S., Peng, S., Peng, Y., and Zhang, X., Three algorithms for solving high-dimensional fully coupled FBSDEs through deep learning, IEEE Intell. Syst., 35 (2020), pp. 71-84. |
| [41] | Ji, S., Peng, S., Peng, Y., and Zhang, X., A Control Method for Solving High-Dimensional Hamiltonian Systems Through Deep Neural Networks, preprint, https://arxiv.org/abs/2111.02636, 2021. |
| [42] | Jiang, Y. and Li, J., Convergence of the Deep BSDE Method for FBSDEs with Non-Lipschitz Coefficients, preprint, https://arxiv.org/abs/2101.01869, 2021. |
| [43] | Kapllani, L. and Teng, L., Deep Learning Algorithms for Solving High Dimensional Nonlinear Backward Stochastic Differential Equations, preprint, https://arxiv.org/abs/2010.01319, 2020. |
| [44] | Kingma, D. P. and Ba, J., Adam: A Method for Stochastic Optimization, preprint, https://arxiv.org/abs/1412.6980, 2014. |
| [45] | Kohatsu-Higa, A., Lejay, A., and Yasuda, K., Weak rate of convergence of the Euler-Maruyama scheme for stochastic differential equations with non-regular drift, J. Comput. Appl. Math., 326 (2017), pp. 138-158. · Zbl 1370.65003 |
| [46] | Kruse, R., Characterization of bistability for stochastic multistep methods, BIT,52 (2012), pp. 109-140. · Zbl 1243.65010 |
| [47] | Liu, Y., Wang, Y., Zhuang, Z., and Guo, X., Deep FBSDE Controller for Attitude Control of Hypersonic Aircraft, in Proceedings of the 6th IEEE International Conference on Advanced Robotics and Mechatronics, , IEEE, 2021, pp. 294-299. |
| [48] | Ma, J., Protter, P., Martín, J. San, and Torres, S., Numerical method for backward stochastic differential equations, Ann. Appl. Probab., 12 (2002), pp. 302-316. · Zbl 1017.60074 |
| [49] | Ma, J., Protter, P., and Yong, J., Solving forward-backward stochastic differential equations explicitly - A four step scheme, Probab. Theory Related Fields, 98 (1994), pp. 339-359. · Zbl 0794.60056 |
| [50] | Mikulevicius, R. and Plate, E., Rate of convergence of the Euler approximation for diffusion processes, Math. Nachr., 151 (1991), pp. 233-239. · Zbl 0733.65104 |
| [51] | Negyesi, B., Andersson, K., and Oosterlee, C. W., The One Step Malliavin scheme: New Discretization of BSDEs Implemented with Deep Learning Regressions, preprint, https://arxiv.org/abs/2110.05421, 2021. |
| [52] | Pereira, M., Wang, Z., Exarchos, I., and Theodorou, E. A., Learning Deep Stochastic Optimal Control Policies Using Forward-Backward SDEs, preprint, https://arxiv.org/abs/1902.03986, 2019. |
| [53] | Raissi, M., Forward-Backward Stochastic Neural Networks: Deep Learning of High-Dimensional Partial Differential Equations, preprint, https://arxiv.org/abs/1804.07010, 2018. |
| [54] | Rockafellar, R. T. and Wets, R. J.-B., Variational Analysis, , Springer Science & Business, Berlin, 2009. |
| [55] | Ruijter, M. J. and Oosterlee, C. W., A Fourier cosine method for an efficient computation of solutions to BSDEs, SIAM J. Sci. Comput., 37 (2015), pp. A859-A889. · Zbl 1314.65011 |
| [56] | Ruijter, M. J. and Oosterlee, C. W., Numerical Fourier method and second-order Taylor scheme for backward SDEs in finance, Appl. Numer. Math., 103 (2016), pp. 1-26. · Zbl 1386.91166 |
| [57] | Teng, L., A review of tree-based approaches to solve forward-backward stochastic differential equations, J. Comput. Finance, 25 (2021), pp. 125-159. |
| [58] | Wang, Y. and Ni, Y.-H., Deep BSDE-ML Learning and Its Application to Model-Free Optimal Control, preprint, https://arxiv.org/abs/2201.01318, 2022. |
| [59] | Yu, Y., Hientzsch, B. and Ganesan, N., Backward Deep BSDE Methods and Applications to Nonlinear Problems, preprint, https://arxiv.org/abs/2006.07635, 2020. |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
