Abstract
The development of robust and agile locomotion skills for legged robots using reinforcement learning is challenging, particularly in demanding environments. In this study, we propose a blind locomotion control learning framework that enables fast and stable walking on challenging terrains. First, we construct an asymmetric terrain feature extraction network that uses a multilayer perceptron to effectively infer terrain features from the history of proprioceptive states, consisting only of inertial measurement unit and joint encoder data. Additionally, our asymmetric actor-critic framework implicitly infers terrain features, thereby enhancing the accuracy of terrain representation. Second, we introduce a foot trajectory generator based on prior gait behaviors, which improves the gait periodicity and provides accurate state information for terrain feature inference. Compared to state-of-the-art methods, our approach significantly increases the learning efficiency by 26.0% and enhances terrain adaptation by 5.0%. It also achieved a more periodic gait, with the state-command tracking error reduced by 38.5% compared with advanced methods. The success rate for traversing complex terrains was similar to that of the baseline methods, with a 31.3% increase in the step height on stair-like terrains. The experimental results demonstrate that the proposed method enables fast and stable walking on challenging terrains.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Availability of data and materials
The datasets generated during the current study are available from the corresponding author upon reasonable request.
References
Lee J, Hwangbo J, Wellhausen L, Koltun V, Hutter M (2020) Learning quadrupedal locomotion over challenging terrain. Sci Robot 5(47):5986
Gehring C, Fankhauser P, Isler L, Diethelm R, Bachmann S, Potz M, Gerstenberg L, Hutter M (2021) Anymal in the field: solving industrial inspection of an offshore hvdc platform with a quadrupedal robot. In: Field and service robotics: results of the 12th international conference, pp 247–260
Biswal P, Mohanty PK (2021) Development of quadruped walking robots: a review. Ain Shams Eng J 12(2):2017–2031
Xu S, Zhu L, Zhang H-T, Ho CP (2023) Robust convex model predictive control for quadruped locomotion under uncertainties. IEEE Trans Rob
Kong NJ, Li C, Council G, Johnson AM (2023) Hybrid ilqr model predictive control for contact implicit stabilization on legged robots. IEEE Trans Rob
Fahmi S, Focchi M, Radulescu A, Fink G, Barasuol V, Semini C (2020) Stance: locomotion adaptation over soft terrain. IEEE Trans Rob 36(2):443–457
Kim Y, Yu B, Lee EM, Kim J-h, Park H-w, Myung H (2022) Step: state estimator for legged robots using a preintegrated foot velocity factor. IEEE Robot Autom Lett 7(2):4456–4463
Jenelten F, Grandia R, Farshidian F, Hutter M (2022) Tamols: terrain-aware motion optimization for legged systems. IEEE Trans Rob 38(6):3395–3413
Winkler AW, Bellicoso CD, Hutter M, Buchli J (2018) Gait and trajectory optimization for legged systems through phase-based end-effector parameterization. IEEE Robot Autom Lett 3(3):1560–1567
Ding Y, Pandala A, Park H-W (2019) Real-time model predictive control for versatile dynamic motions in quadrupedal robots. In: 2019 International Conference on Robotics and Automation (ICRA), pp 8484–8490
Green K, Godse Y, Dao J, Hatton RL, Fern A, Hurst J (2021) Learning spring mass locomotion: guiding policies with a reduced-order model. IEEE Robot Autom Lett 6(2):3926–3932
Kumar A, Li Z, Zeng J, Pathak D, Sreenath K, Malik J (2022) Adapting rapid motor adaptation for bipedal robots. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 1161–1168
Zhang H, Wang J, Wu Z, Wang Y, Wang D (2021) Terrain-aware risk-assessment-network-aided deep reinforcement learning for quadrupedal locomotion in tough terrain. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 4538–4545
Imai CS, Zhang M, Zhang Y, Kierebiński M, Yang R, Qin Y, Wang X (2022) Vision-guided quadrupedal locomotion in the wild with multi-modal delay randomization. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 5556–5563
Miki T, Lee J, Hwangbo J, Wellhausen L, Koltun V, Hutter M (2022) Learning robust perceptive locomotion for quadrupedal robots in the wild. Sci Robot 7(62):2822
Agarwal A, Kumar A, Malik J, Pathak D (2023) Legged locomotion in challenging terrains using egocentric vision. In: Conference on robot learning, pp 403–415
Sorokin M, Tan J, Liu CK, Ha S (2022) Learning to navigate sidewalks in outdoor environments. IEEE Robot Autom Lett 7(2):3906–3913
Lim H, Oh M, Myung H (2021) Patchwork: concentric zone-based region-wise ground segmentation with ground likelihood estimation using a 3d lidar sensor. IEEE Robot Autom Lett 6(4):6458–6465
Lee S, Lim H, Myung H (2022) Patchwork++: fast and robust ground segmentation solving partial under-segmentation using 3d point cloud. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 13276–13283
Choi S, Ji G, Park J, Kim H, Mun J, Lee JH, Hwangbo J (2023) Learning quadrupedal locomotion on deformable terrain. Sci Robot 8(74):2256
Kumar A, Fu Z, Pathak D, Malik J (2021) Rma: rapid motor adaptation for legged robots. Robotics: Science and Systems XVII
Ji G, Mun J, Kim H, Hwangbo J (2022) Concurrent training of a control policy and a state estimator for dynamic and robust legged locomotion. IEEE Robot Autom Lett 7(2):4630–4637
Nahrendra IMA, Yu B, Myung H (2023) Dreamwaq: learning robust quadrupedal locomotion with implicit terrain imagination via deep reinforcement learning. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), pp 5078–5084
Luo Z, Dong Y, Li X, Huang R, Shu Z, Xiao E, Lu P (2024) Moral: learning morphologically adaptive locomotion controller for quadrupedal robots on challenging terrains. IEEE Robot Autom Lett 9(5):4019–4026
Pinto L, Andrychowicz M, Welinder P, Zaremba W, Abbeel P (2018) Asymmetric actor critic for image-based robot learning. In: 14th Robotics: Science and Systems, RSS 2018
Andrychowicz OM, Baker B, Chociej M, Jozefowicz R, McGrew B, Pachocki J, Petron A, Plappert M, Powell G, Ray A et al (2020) Learning dexterous in-hand manipulation. Int J Robot Res 39(1):3–20
Rashid T, Samvelyan M, De Witt CS, Farquhar G, Foerster J, Whiteson S (2020) Monotonic value function factorisation for deep multi-agent reinforcement learning. J Mach Learn Res 21(178):1–51
Wu J, Xin G, Qi C, Xue Y (2023) Learning robust and agile legged locomotion using adversarial motion priors. IEEE Robot Autom Lett
Margolis GB, Yang G, Paigwar K, Chen T, Agrawal P (2024) Rapid locomotion via reinforcement learning. Int J Robot Res 43(4):572–587
Peng XB, Abbeel P, Levine S, Panne M (2018) Deepmimic: example-guided deep reinforcement learning of physics-based character skills. ACM Trans Graph (TOG) 37(4):1–14
Shi H, Zhou B, Zeng H, Wang F, Dong Y, Li J, Wang K, Tian H, Meng MQ-H (2022) Reinforcement learning with evolutionary trajectory generator: a general approach for quadrupedal locomotion. IEEE Robot Autom Lett 7(2):3085–3092
Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347
Rudin N, Hoeller D, Reist P, Hutter M (2022) Learning to walk in minutes using massively parallel deep reinforcement learning. In: Conference on robot learning, pp 91–100
Escontrela A, Peng XB, Yu W, Zhang T, Iscen A, Goldberg K, Abbeel P (2022) Adversarial motion priors make good substitutes for complex reward functions. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 25–32
Hwangbo J, Lee J, Dosovitskiy A, Bellicoso D, Tsounis V, Koltun V, Hutter M (2019) Learning agile and dynamic motor skills for legged robots. Sci Robot 4(26):5872
Fu Z, Kumar A, Malik J, Pathak D (2022) Minimizing energy consumption leads to the emergence of gaits in legged robots. In: Conference on robot learning, pp 928–937
Xie Z, Ling HY, Kim NH, Panne M (2020) Allsteps: curriculum-driven learning of stepping stone skills. Comput Graphics Forum 39
Muzio AF, Maximo MR, Yoneyama T (2022) Deep reinforcement learning for humanoid robot behaviors. J Intell Robot Syst 105(1):12
Bonardi A, James S, Davison AJ (2020) Learning one-shot imitation from humans without humans. IEEE Robot Autom Lett 5(2):3533–3539
Zhu W, Guo X, Owaki D, Kutsuzawa K, Hayashibe M (2021) A survey of sim-to-real transfer techniques applied to reinforcement learning for bioinspired robots. IEEE Trans Neural Netw Learn Syst 34(7):3444–3459
Acknowledgements
Funding from the National Nature Science Foundation of China and the Open Projects Program of the State Key Laboratory of Multimodal Artificial Intelligence Systems are gratefully acknowledged
Funding
This work was supported by the National Nature Science Foundation of China (62373016, 61873008) and the Open Projects Program of the State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS-2023-22).
Author information
Authors and Affiliations
Contributions
In this work, Guoyu Zuo first proposed the research idea and approach of the paper and provided project guidance. The construction of the framework and the experimental platform was completed by Guoyu Zuo and Yong Wang. The code implementation, development, experiment testing, and data analysis were done by Yong Wang. The first draft of the manuscript was written by Yong Wang and Shuangyue Yu. Guoyu Zuo, Yong Wang, Daoxiong Gong, and Shuangyue Yu provided valuable suggestions and feedback on the draft. Guoyu Zuo, Daoxiong Gong, and Shuangyue Yu also made some important revisions to the final paper. All authors contributed to the article and approved the submitted version.
Corresponding author
Ethics declarations
Ethics approval
The authors declare that they have no conflict of interest. This paper has not been previously published, it is published with the permission of the authors’ institution, and all authors of this paper are responsible for the authenticity of the data in the paper.
Consent to participate
All authors of this paper have been informed of the revision and publication of the paper, have checked all data, figures, and tables in the manuscript, and are responsible for their truthfulness and accuracy. Names of all contributing authors: Guoyu Zuo, Yong Wang, Shuangyue Yu.
Consent for publication
The publication has been approved by all co-authors.
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zuo, G., Wang, Y., Gong, D. et al. Learning quadrupedal locomotion on tough terrain using an asymmetric terrain feature mining network. Appl Intell 54, 11547–11563 (2024). https://doi.org/10.1007/s10489-024-05782-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-024-05782-7