Article Contents

2024, Volume 11, Issue 3: 249-264. Doi: 10.3934/jdg.2023025

This issue Previous Article DSSA: Direct Simplified Symbolic analysis using metaheuristic-driven circuit modelling Next Article Discrete-time nonstationary average stochastic games

The relationships between discounted and average criteria of stochastic games with prospect theory

Yiting Wu^1, and
Junyu Zhang^1, ,

1.
School of Mathematics, Sun Yat-Sen University, Guangzhou 510275, China

^*Corresponding author: Junyu Zhang
^*Corresponding author: Junyu Zhang

Received: October 2023

Revised: December 2023

Early access: December 2023

Published: July 2024

Abstract / Introduction Full Text(HTML) Related Papers Cited by

Abstract

This paper considers nonzero-sum discrete-time stochastic games with finite state and action spaces, and focuses on the performance criteria under prospect theory. Based on the average criterion of stochastic games with prospect theory established firstly in 2018, we first present the discounted criterion of stochastic games with prospect theory and study the relationships between them. Both the criterion/value function and the Nash equilibrium are studied. We derive the equality of discounted and average criterion functions for any fixed strategy by the Abelian theorem, and discuss the relationships of value function. Moreover, since the probability is distorted in prospect theory, there is no optimality equation which is the commonly used tool in the existing literature of Markov decision processes and stochastic games. In this work, the bilateral relationships of equilibria between discounted and average criterion are investigated by the performance criterion (instead of the optimality equation), with the convergence of strategy sequences.

Keywords:

Mathematics Subject Classification: Primary: 91A10, 91A15; Secondary: 91A50.

Citation:

Full Text(HTML)

Related Papers

Cited by

References

[1]	E. Altman, A. Hordijk and F. M. Spieksma, Contraction conditions for average and $\alpha$-discount optimality in countable state Markov games with unbounded rewards, Mathematics of Operations Research, 22 (1997), 588-618. doi: 10.1287/moor.22.3.588.
[2]	R. Blancas-Rivera, R. Cavazos-Cadena and H. Cruz-Suárez, Discounted approximations in risk-sensitive average Markov cost chains with finite state space, Mathematical Methods of Operations Research, 91 (2020), 241-268. doi: 10.1007/s00186-019-00689-3.
[3]	M. Bourque and T. E. S. Raghavan, Policy improvement for perfect information additive reward and additive transition stochastic games with discounted and average payoffs, Journal of Dynamics and Games, 1 (2014), 347-361. doi: 10.3934/jdg.2014.1.347.
[4]	R. Cavazos-Cadena and D. Hernández-Hernández, Vanishing discount approximations in controlled Markov chains with risk-sensitive average criterion, Advances in Applied Probability, 50 (2018), 204-230. doi: 10.1017/apr.2018.10.
[5]	R. Cavazos-Cadena and D. Hernández-Hernández, The vanishing discount approach in a class of zero-sum finite games with risk-sensitive average criterion, SIAM Journal on Control and Optimization, 57 (2019), 219-240. doi: 10.1137/18M1165104.
[6]	R. Cavazos-Cadena and F. Salem-Silva, The discounted method and equivalence of average criteria for risk-sensitive Markov decision processes on Borel spaces, Applied Mathematics & Optimization, 61 (2010), 167-190. doi: 10.1007/s00245-009-9080-2.
[7]	R. Dekker and A. Hordijk, Average, sensitive and Blackwell optimal policies in denumerable Markov decision chains with unbounded rewards, Mathematics of Operations Research, 13 (1988), 395-420. doi: 10.1287/moor.13.3.395.
[8]	J. Duggan, Noisy stochastic games, Econometrica, 80 (2012), 2017-2045. doi: 10.3982/ECTA10125.
[9]	B. A. Escobedo-Trujillo, A. Alaffita-Hernández and R. López-Martínez, Constrained stochastic differential games with additive structure: Average and discount payoffs, Journal of Dynamics and Games, 5 (2018), 109-141. doi: 10.3934/jdg.2018008.
[10]	S. R. Etesami, W. Saad, N. B. Mandayam and H. V. Poor, Stochastic games for the smart grid energy management with prospect prosumers, IEEE Transactions on Automatic Control, 63 (2018), 2327-2342. doi: 10.1109/tac.2018.2797217.
[11]	A. Federgruen, On $N$-person stochastic games by denumerable state space, Advances in Applied Probability, 10 (1978), 452-471. doi: 10.2307/1426945.
[12]	E. A. Feinberg and A. Shwartz, Handbook of Markov Decision Processes: Methods and Application, Kluwer Academic Publishers, Boston, 2002. doi: 10.1007/978-1-4615-0805-2.
[13]	J. Filar and K. Vrieze, Competitive Markov Decision Processes, Springer-Verlag, New York, 1997. doi: 10.1007/978-1-4612-4054-9.
[14]	D. Goreac and O.-S. Serea, Abel-type results for controlled piecewise deterministic Markov processes, Differential Equations and Dynamical Systems, 25 (2017), 83-100. doi: 10.1007/s12591-015-0245-y.
[15]	O. Hernández-Lerma and J. B. Lasserre, Discrete-Time Markov Control Processes: Basic Optimality Criteria, Springer-Verlag, New York, 1996. doi: 10.1007/978-1-4612-0729-0.
[16]	D. Kahneman and A. Tversky, Prospect theory: An analysis of decision under risk, Econometrica, 47 (1979), 263-291. doi: 10.2307/1914185.
[17]	M. N. Katehakis and U. G. Rothblum, Finite state multi-armed bandit problems: Sensitive-discount, average-reward and average-overtaking optimality, The Annals of Applied Probability, 6 (1996), 1024-1034. doi: 10.1214/aoap/1034968239.
[18]	S. Lakshminarayana, T. Q. S. Quek and H. V. Poor, Cooperation and storage tradeoffs in power grids with renewable energy resources, IEEE Journal on Selected Areas in Communications, 32 (2014), 1386-1397. doi: 10.1109/JSAC.2014.2332093.
[19]	K. Lin, Stochastic Systems with Cumulative Prospect Theory, Ph.D thesis, University of Maryland, College Park, 2013.
[20]	Q. Liu and X. Zou, A risk minimization problem for finite horizon semi-Markov decision processes with loss rates, Journal of Dynamics and Games, 5 (2018), 143-163. doi: 10.3934/jdg.2018009.
[21]	J.-F. Mertens, Stochastic games, Handbook of Game Theory with Economic Applications, Elsevier Science B.V., North Holland, Amsterdam, 3 (2002), 1809-1832. doi: 10.1016/S1574-0005(02)03010-2.
[22]	A. Neyman and S. Sorin, Stochastic Games and Applications, Kluwer Academic Publishers, Dordrecht, 2003. doi: 10.1007/978-94-010-0189-2.
[23]	C. Pal and S. Pradhan, Zero-sum games for pure jump processes with risk-sensitive discounted cost criteria, Journal of Dynamics and Games, 9 (2022), 13-25. doi: 10.3934/jdg.2021020.
[24]	L. I. Sennott, Average cost optimal stationary policies in infinite state Markov decision processes with unbounded costs, Operations Research, 37 (1989), 626-633. doi: 10.1287/opre.37.4.626.
[25]	L. I. Sennott, Stochastic Dynamic Programming and the Control of Queueing Systems, Wiley, New Jersey, 1999. doi: 10.1002/9780470317037.
[26]	M. J. Sobel, Noncooperative stochastic games, The Annals of Mathematical Statistics, 42 (1971), 1930-1935. doi: 10.1214/aoms/1177693059.
[27]	A. F. Veinott Jr., Discrete dynamic programming with sensitive discount optimality criteria, Annals of Mathematical Statistics, 40 (1969), 1635-1660. doi: 10.1214/aoms/1177697379.
[28]	Q. Wei and X. Chen, Nonzero-sum expected average discrete-time stochastic games: The case of uncountable spaces, SIAM Journal on Control and Optimization, 57 (2019), 4099-4124. doi: 10.1137/19M1248509.
[29]	Y. Wu and J. Zhang, Discounted stochastic games for the smart grid with prospect prosumers, work in progress.