Distributed event-triggered optimal bipartite consensus control for multiagent systems with input delay via reinforcement learning method. (English) Zbl 07892626
Summary: In this article, an optimal bipartite consensus control (OBCC) scheme is proposed for heterogeneous multiagent systems (MASs) with input delay by reinforcement learning (RL) algorithm. A directed signed graph is established to construct MASs with competitive and cooperative relationships, and model reduction method is developed to tackle input delay problem. Then, based on the Hamilton-Jacobi-Bellman (HJB) equation, policy iteration method is utilized to design the bipartite consensus controller, which consists of value function and optimal controller. Further, a distributed event-triggered function is proposed to increase control efficiency, which only requires information from its own agent and neighboring agents. Based on the input-to-state stability (ISS) function and Lyapunov function, sufficient conditions for the stability of MASs can be derived. Apart from that, RL algorithm is employed to solve the event-triggered OBCC problem in MASs, where critic neural networks (NNs) and actor NNs estimate value function and control policy, respectively. Finally, simulation results are given to validate the feasibility and efficiency of the proposed algorithm.
© 2023 Chinese Automatic Control Society and John Wiley & Sons Australia, Ltd
© 2023 Chinese Automatic Control Society and John Wiley & Sons Australia, Ltd
MSC:
| 93C55 | Discrete-time control/observation systems |
| 93A16 | Multi-agent systems |
| 93D50 | Consensus |
| 93B70 | Networked control |
Keywords:
coopetition MASs; distributed event-triggered control; model reduction method; optimal bipartite consensus control; reinforcement learningReferences:
| [1] | R.Yang, L.Liu, and G.Feng, An overview of recent advances in distributed coordination of multi‐agent systems, Unmanned Syst.10 (2022), no. 03, 307-325. |
| [2] | J.Qin, Q.Ma, Y.Shi, and L.Wang, Recent advances in consensus of multi‐agent systems: A brief survey, IEEE Trans. Ind. Electron.64 (2017), no. 6, 4972-4983. |
| [3] | H.Zhang, H.Jiang, Y.Luo, and G.Xiao, Data‐driven optimal consensus control for discrete‐time multi‐agent systems with unknown dynamics using reinforcement learning method, IEEE Trans. Ind. Electron.64 (2017), no. 5, 4091-4100. |
| [4] | X.Dong, B.Yu, Z.Shi, and Y.Zhong, Time‐varying formation control for unmanned aerial vehicles: Theories and applications, IEEE Trans. Control Syst. Technol.23 (2015), no. 1, 340-348. |
| [5] | J.Ma, M.Ye, Y.Zheng, and Y.Zhu, Consensus analysis of hybrid multiagent systems: A game‐theoretic approach, Int. J. Robust Nonlinear Control29 (2019), no. 6, 1840-1853. · Zbl 1416.93184 |
| [6] | A.Liu, W. A.Zhang, L.Yu, H.Yan, and R.Zhang, Formation control of multiple mobile robots incorporating an extended state observer and distributed model predictive approach, IEEE Trans. Syst., Man, Cybern.: Syst.50 (2020), no. 11, 4587-4597. |
| [7] | A.Albert‐Cromarias and C.Dos Santos, Coopetition in healthcare: Heresy or reality? An exploration of felt outcomes at an intra‐organizational level, Social Sci. Med.252 (2020), 112938. |
| [8] | S.Manzhynski and F.Figge, Coopetition for sustainability: Between organizational benefit and societal good, Bus. Strat. Environ.29 (2020), no. 3, 827-837. |
| [9] | H.Yu, X.Chen, T.Chen, and F.Hao, Event‐triggered bipartite consensus for multiagent systems: A Zeno‐free analysis, IEEE Trans. Autom. Control65 (2020), 4866-4873. · Zbl 1536.93587 |
| [10] | C.Altafini, Consensus problems on networks with antagonistic interactions, IEEE Trans. Autom. Control58 (2013), no. 4, 935-946. · Zbl 1369.93433 |
| [11] | J. A.Guerrero, D.Olivares, and G.Romero, Forced bipartite consensus for multi‐agent systems, IECON2018 ‐ 44th Annual Conference of the IEEE Industrial Electronics Society, 2018, pp. 2335-2340. |
| [12] | J.Lu, Y.Wang, X.Shi, and J.Cao, Finite‐time bipartite consensus for multiagent systems under detail‐balanced antagonistic interactions, IEEE Trans. Syst., Man, Cybern.: Syst.51 (2021), no. 6, 3867-3875. |
| [13] | Y.Cai, H.Zhang, J.Duan, and J.Zhang, Distributed bipartite consensus of linear multiagent systems based on event‐triggered output feedback control scheme, IEEE Trans. Syst., Man, Cybern.: Syst.51 (2021), no. 11, 6743-6756. |
| [14] | X.Nan, Y.Lv, and Z.Duan, Bipartite consensus tracking for antagonistic topologies with leader’s unknown input, Asian J. Control24 (2022), no. 2, 547-561. · Zbl 07886994 |
| [15] | X.Ai, Adaptive robust bipartite consensus of high‐order uncertain multi‐agent systems over cooperation‐competition networks, J. Franklin Inst.357 (2020), no. 3, 1813-1831. · Zbl 1430.93188 |
| [16] | H.Zhang, J.Duan, Y.Wang, and Z.Gao, Bipartite fixed‐time output consensus of heterogeneous linear multiagent systems, IEEE Trans. Cybern.51 (2021), no. 2, 548-557. |
| [17] | K. G.Vamvoudakis, F. L.Lewis, and G. R.Hudas, Multi‐agent differential graphical games: Online adaptive learning solution for synchronization with optimality, Automatica48 (2012), no. 8, 1598-1611. · Zbl 1267.93190 |
| [18] | X.Yang and B.Li, Optimized tracking control using reinforcement learning strategy for a class of nonlinear systems, Asian J. Control25 (2022), 2095-2104. · Zbl 07889134 |
| [19] | M.Abouheaf, F.Lewis, M.Mahmoud, and D.Mikulski, Discrete‐time dynamic graphical games: Model‐free reinforcement learning solution, Control Theory Technol.13 (2015), 55-69. · Zbl 1340.91018 |
| [20] | G.Jing, H.Bai, J.George, A.Chakrabortty, and P. K.Sharma, Distributed cooperative multi‐agent reinforcement learning with directed coordination graph, 2022. ArXiv abs/2201.04962. |
| [21] | G.Wen, C. L. P.Chen, and B.Li, Optimized formation control using simplified reinforcement learning for a class of multiagent systems with unknown dynamics, IEEE Trans. Ind. Electron.67 (2020), no. 9, 7879-7888. |
| [22] | Z.Peng, J.Hu, K.Shi, R.Luo, R.Huang, B. K.Ghosh, and J.Huang, A novel optimal bipartite consensus control scheme for unknown multi‐agent systems via model‐free reinforcement learning, Appl. Math. Comput.369 (2020), 124821. · Zbl 1433.93008 |
| [23] | Q.Li, L.Xia, R.Song, and J.Liu, Leader follower bipartite output synchronization on signed digraphs under adversarial factors via data‐based reinforcement learning, IEEE Trans. Neural Netw. Learn. Syst.31 (2020), no. 10, 4185-4195. |
| [24] | J.Ren, Q.Song, Y.Gao, and G.Lu, Leader‐following bipartite consensus of second‐order time‐delay nonlinear multi‐agent systems with event‐triggered pinning control under signed digraph, Neurocomputing385 (2020), 186-196. |
| [25] | T.‐F.Ding, M.‐F.Ge, C.‐H.Xiong, J. H.Park, and M.Li, Second‐order bipartite consensus for networked robotic systems with quantized‐data interactions and time‐varying transmission delays, ISA Trans.108 (2021), 178-187. |
| [26] | Y.Cai, H.Zhang, J.Zhang, and Q.He, Distributed bipartite leader‐following consensus of linear multi‐agent systems with input time delay based on event‐triggered transmission mechanism, ISA Trans.100 (2020), 221-234. |
| [27] | J.Zhang, H.Ma, W.Li, and Y.Zhang, Leader‐follower optimal bipartite consensus control for multi‐agent systems with time‐delay using reinforcement learning method, Proceedings ‐ 2020 Chinese Automation Congress, CAC 2020, Shanghai, China, 2020, pp. 1587-1592. |
| [28] | S.Hu, X.Chen, J.Qiu, F.Zhao, X.Jiang, and Y.Du, Dynamic event‐triggered bipartite consensus of multiagent systems with estimator and cooperative‐competitive interactions, IEEE Trans. Circ. Syst. II: Express Briefs69 (2022), no. 7, 3309-3313. |
| [29] | Z.Wang, Q.Wei, and D.Liu, A novel triggering condition of event‐triggered control based on heuristic dynamic programming for discrete‐time systems, Opt. Control Appl. Methods39 (2018), no. 4, 1467-1478. · Zbl 1398.93224 |
| [30] | L.Rong, Y.Hua, and G.‐P.Jiang, Distributed consensus control for networked linear agents via event‐triggered communications, Asian J. Control24 (2022), no. 2, 537-546. · Zbl 07886993 |
| [31] | G.Zhao, H.Wei, and X.Fu, Distributed dynamic event‐triggered control approach to multi‐agent systems via adaptive consensus protocols, Asian J. Control24 (2022), no. 3, 1486-1496. · Zbl 07887069 |
| [32] | X.Wang, D.Ding, X.Ge, and Q.‐L.Han, Neural‐network‐based control for discrete‐time nonlinear systems with denial‐of‐service attack: The adaptive event‐triggered case, Int. J. Robust Nonlinear Control32 (2022), no. 5, 2760-2779. · Zbl 1527.93298 |
| [33] | H.Liang, X.Guo, Y.Pan, and T.Huang, Event‐triggered fuzzy bipartite tracking control for network systems based on distributed reduced‐order observers, IEEE Trans. Fuzzy Syst.29 (2021), no. 6, 1601-1614. |
| [34] | Y.Xu, J.Wang, Y.Zhang, and Y.Xu, Event‐triggered bipartite consensus for high‐order multi‐agent systems with input saturation, Neurocomputing379 (2020), 284-295. |
| [35] | X.Wang, D.Ding, X.Ge, and Q.‐L.Han, Supplementary control for quantized discrete‐time nonlinear systems under goal representation heuristic dynamic programming, IEEE Trans. Neural Netw. Learn. Syst.2022 (2022), 1-13. |
| [36] | M.Fu and L.Wang, Event‐triggered‐based finite‐time cooperative path following control for underactuated multiple marine surface vehicles, Asian J. Control25 (2023), no. 2, 1005-1018. · Zbl 07889053 |
| [37] | X.Yang and H.He, Decentralized event‐triggered control for a class of nonlinear‐interconnected systems using reinforcement learning, IEEE Trans. Cybern.51 (2021), no. 2, 635-648. |
| [38] | H.Li, Y.Wu, M.Chen, and R.Lu, Adaptive multigradient recursive reinforcement learning event‐triggered tracking control for multiagent systems, IEEE Trans. Neural Netw. Learn. Syst.34 (2023), no. 1, 144-156. |
| [39] | J.Li and X.Chen, Event‐triggered control for bipartite consensus and formation of multiple wheeled mobile robots with non‐holonomic constraint, Asian J. Control24 (2022), no. 4, 1795-1807. · Zbl 07887095 |
| [40] | S.Hu, X.Chen, J.Qiu, F.Zhao, X.Jiang, and Y.Du, Dynamic event‐triggered bipartite consensus of multiagent systems with estimator and cooperative‐competitive interactions, IEEE Trans. Circ. Syst. II‐Express Briefs69 (2022), no. 7, 3309-3313. |
| [41] | X.Chen, H.Yu, and F.Hao, Prescribed‐time event‐triggered bipartite consensus of multiagent systems, IEEE Trans. Cybern.52 (2022), no. 4, 2589-2598. |
| [42] | H.‐W.Liu, S.‐L.Du, X.‐F.Wang, T.Sun, and C.‐Q.Zhong, Consensus of multiagent systems with time‐varying delays: An observer‐based distributed periodic event‐triggered control approach, Asian J. Control24 (2022), no. 2, 712-721. · Zbl 07887006 |
| [43] | W.Wang, X.Chen, H.Fu, and M.Wu, Model‐free distributed consensus control based on actor‐critic framework for discrete‐time nonlinear multiagent systems, IEEE Trans. Syst. Man Cybern.‐Syst.50 (2020), no. 11, 4123-4134. |
| [44] | H.Zhang, D.Yue, C.Dou, W.Zhao, and X.Xie, Data‐driven distributed optimal consensus control for unknown multiagent systems with input‐delay, IEEE Trans. Cybern.49 (2019), no. 6, 2095-2105. |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
