This paper deals with the risk probability for finite horizon semi-Markov decision processes with loss rates. The criterion to be minimized is the risk probability that the total loss incurred during a finite horizon exceed a loss level. For such an optimality problem, we first establish the optimality equation, and prove that the optimal value function is a unique solution to the optimality equation. We then show the existence of an optimal policy, and develop a value iteration algorithm for computing the value function and optimal policies. We also derive the approximation of the value function and the rules of iteration. Finally, a numerical example is given to illustrate our results.
Citation: |
N. Bauerle and U. Rieder, Markov Decision Processes with Application to Finance, Universitext, Springer, Heidelberg, 2011. | |
M. Bouakiz and Y. Kebir , Target-level criterion in Markov decision processes, Journal of Optimization Theory and Applications, 86 (1995) , 1-15. doi: 10.1007/BF02193458. | |
M. K. Ghosh and S. Subhamay , Non-stationary semi-Markov secision processes on a finite horizon, Stochastic Analysis and Applications, 31 (2013) , 183-190. doi: 10.1080/07362994.2013.741405. | |
X. P. Guo and O. Hernández-Lerma, Continuous-Time Markov Decision Processes: Theory and Applications, Springer-Verlag, Berlin, 2009. | |
X. P. Guo and J. Yang , A new condition and approach for zero-sum stochastic games with average payoffs, Stochastic Analysis and Applications, 26 (2008) , 537-561. doi: 10.1080/07362990802007095. | |
X. P. Guo , P. Shi and W. P. Zhu , Strong average optimality for controlled nonhomogeneous Markov chains, Stochastic Analysis and Applications, 19 (2001) , 115-134. doi: 10.1081/SAP-100001186. | |
O. Hernández-Lerma and J. B. Lasserre, Discrete-time Markov Control Processes, Basic optimality criteria, Springer-Verlag, New York, 1996. | |
O. Hernández-Lerma and J. B. Lasserre, Further Topics on Discrete-Time Markov Control Processes, Springer-Verlag, New York, 1999. | |
Y. H. Huang and X. P. Guo , Optimal risk probability for first passage models in semi-Markov decision processes, Journal Mathematical Analysis Applications, 359 (2009) , 404-420. doi: 10.1016/j.jmaa.2009.05.058. | |
Y. H. Huang , X. P. Guo and X. Y. Song , Performance analysis for controlled semi-Markov systems with application to maintenance, Journal of Optimization Theory and Applications, 150 (2011) , 395-415. doi: 10.1007/s10957-011-9813-7. | |
Y. H. Huang and X. P. Guo , Finite horizon semi-Markov decision processes with application to maintenance systems, European Journal Operations Research, 212 (2011) , 131-140. doi: 10.1016/j.ejor.2011.01.027. | |
Y. H. Huang , X. P. Guo and Z. F. Li , Minimum risk probability for finite horizon semi-Markov decision processes, Journal Mathematical Analysis Applications, 402 (2013) , 378-391. doi: 10.1016/j.jmaa.2013.01.021. | |
Y. H. Huang and X. P. Guo , Mean-variance problems for finite horizon semi-Markov decision processes, Applications Mathematical Optimization, 72 (2015) , 233-259. doi: 10.1007/s00245-014-9278-9. | |
N. Limnios and G. Oprisan, Semi-Markov Processes and Reliability, Birkhäuser Boston, Inc., Boston, MA, 2001. | |
J. Y. Liu and S. M. Huang , Markov decision processes with distribution function criterion of first-passage time, Applications Mathematical Optimization, 43 (2001) , 187-201. doi: 10.1007/s00245-001-0007-9. | |
P. M. Madhani , Rebalancing fixed and variable pay in a sales organization: A business cycle perspective, Compensation Benefits Review, 42 (2010) , 179-189. doi: 10.1177/0886368709359668. | |
J. W. Mamer , Successive approximations for finite horizon semi-Markov decision processes with application to asset liquidation, Oper. Res., 34 (1986) , 638-644. doi: 10.1287/opre.34.4.638. | |
Y. Ohtsubo , Minimizing risk models in stochastic shortest path problems, Mathematical Methods of Operations Research, 57 (2003) , 79-88. doi: 10.1007/s001860200246. | |
Y. Ohtsubo , Optimal threshold probability in undiscounted Markov decision processes with a target set, Appl. Math. Comput., 149 (2004) , 519-532. doi: 10.1016/S0096-3003(03)00158-9. | |
M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons, Inc., New York, 1994. | |
C. Ruhm , Are recessions good for your health?, Quarterly Journal of Economics, 115 (2000) , 617-650. doi: 10.3386/w5570. | |
M. Sakaguchi and Y. Ohtsubo , Markov decision processes associated with two threshold probability criteria, Journal Control Theory Applications, 11 (2013) , 548-557. doi: 10.1007/s11768-013-2194-8. | |
Q. D. Wei and X. P. Guo , New average optimality conditions for semi-Markov decision processes in Borel spaces, Journal of Optimization Theory and Applications, 153 (2012) , 709-732. doi: 10.1007/s10957-012-9986-8. | |
D. J. White , Minimising a threshold probability in discounted Markov decision processes, J. Math. Anal. Appl., 173 (1993) , 634-646. doi: 10.1006/jmaa.1993.1093. | |
Y. H. Wu , Bounds for the ruin probability under a Markovian modulated risk model, Communications in statistics Stochastic Models, 15 (1999) , 125-136. doi: 10.1080/15326349908807529. | |
S. X. Yu , Y. L. Lin and P. F. Yan , Optimization models for the first arrival target distribution function in discrete time, J. Math. Anal. Appl., 225 (1998) , 193-223. doi: 10.1006/jmaa.1998.6015. |
The function
The function
The function
The function
The function