Skip to main content
Springer Nature Link
Log in

Learning quadrupedal locomotion on tough terrain using an asymmetric terrain feature mining network

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The development of robust and agile locomotion skills for legged robots using reinforcement learning is challenging, particularly in demanding environments. In this study, we propose a blind locomotion control learning framework that enables fast and stable walking on challenging terrains. First, we construct an asymmetric terrain feature extraction network that uses a multilayer perceptron to effectively infer terrain features from the history of proprioceptive states, consisting only of inertial measurement unit and joint encoder data. Additionally, our asymmetric actor-critic framework implicitly infers terrain features, thereby enhancing the accuracy of terrain representation. Second, we introduce a foot trajectory generator based on prior gait behaviors, which improves the gait periodicity and provides accurate state information for terrain feature inference. Compared to state-of-the-art methods, our approach significantly increases the learning efficiency by 26.0% and enhances terrain adaptation by 5.0%. It also achieved a more periodic gait, with the state-command tracking error reduced by 38.5% compared with advanced methods. The success rate for traversing complex terrains was similar to that of the baseline methods, with a 31.3% increase in the step height on stair-like terrains. The experimental results demonstrate that the proposed method enables fast and stable walking on challenging terrains.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Availability of data and materials

The datasets generated during the current study are available from the corresponding author upon reasonable request.

References

  1. Lee J, Hwangbo J, Wellhausen L, Koltun V, Hutter M (2020) Learning quadrupedal locomotion over challenging terrain. Sci Robot 5(47):5986

    Article  Google Scholar 

  2. Gehring C, Fankhauser P, Isler L, Diethelm R, Bachmann S, Potz M, Gerstenberg L, Hutter M (2021) Anymal in the field: solving industrial inspection of an offshore hvdc platform with a quadrupedal robot. In: Field and service robotics: results of the 12th international conference, pp 247–260

  3. Biswal P, Mohanty PK (2021) Development of quadruped walking robots: a review. Ain Shams Eng J 12(2):2017–2031

    Article  Google Scholar 

  4. Xu S, Zhu L, Zhang H-T, Ho CP (2023) Robust convex model predictive control for quadruped locomotion under uncertainties. IEEE Trans Rob

  5. Kong NJ, Li C, Council G, Johnson AM (2023) Hybrid ilqr model predictive control for contact implicit stabilization on legged robots. IEEE Trans Rob

  6. Fahmi S, Focchi M, Radulescu A, Fink G, Barasuol V, Semini C (2020) Stance: locomotion adaptation over soft terrain. IEEE Trans Rob 36(2):443–457

    Article  Google Scholar 

  7. Kim Y, Yu B, Lee EM, Kim J-h, Park H-w, Myung H (2022) Step: state estimator for legged robots using a preintegrated foot velocity factor. IEEE Robot Autom Lett 7(2):4456–4463

    Article  Google Scholar 

  8. Jenelten F, Grandia R, Farshidian F, Hutter M (2022) Tamols: terrain-aware motion optimization for legged systems. IEEE Trans Rob 38(6):3395–3413

    Article  Google Scholar 

  9. Winkler AW, Bellicoso CD, Hutter M, Buchli J (2018) Gait and trajectory optimization for legged systems through phase-based end-effector parameterization. IEEE Robot Autom Lett 3(3):1560–1567

    Article  Google Scholar 

  10. Ding Y, Pandala A, Park H-W (2019) Real-time model predictive control for versatile dynamic motions in quadrupedal robots. In: 2019 International Conference on Robotics and Automation (ICRA), pp 8484–8490

  11. Green K, Godse Y, Dao J, Hatton RL, Fern A, Hurst J (2021) Learning spring mass locomotion: guiding policies with a reduced-order model. IEEE Robot Autom Lett 6(2):3926–3932

    Article  Google Scholar 

  12. Kumar A, Li Z, Zeng J, Pathak D, Sreenath K, Malik J (2022) Adapting rapid motor adaptation for bipedal robots. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 1161–1168

  13. Zhang H, Wang J, Wu Z, Wang Y, Wang D (2021) Terrain-aware risk-assessment-network-aided deep reinforcement learning for quadrupedal locomotion in tough terrain. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 4538–4545

  14. Imai CS, Zhang M, Zhang Y, Kierebiński M, Yang R, Qin Y, Wang X (2022) Vision-guided quadrupedal locomotion in the wild with multi-modal delay randomization. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 5556–5563

  15. Miki T, Lee J, Hwangbo J, Wellhausen L, Koltun V, Hutter M (2022) Learning robust perceptive locomotion for quadrupedal robots in the wild. Sci Robot 7(62):2822

    Article  Google Scholar 

  16. Agarwal A, Kumar A, Malik J, Pathak D (2023) Legged locomotion in challenging terrains using egocentric vision. In: Conference on robot learning, pp 403–415

  17. Sorokin M, Tan J, Liu CK, Ha S (2022) Learning to navigate sidewalks in outdoor environments. IEEE Robot Autom Lett 7(2):3906–3913

    Article  Google Scholar 

  18. Lim H, Oh M, Myung H (2021) Patchwork: concentric zone-based region-wise ground segmentation with ground likelihood estimation using a 3d lidar sensor. IEEE Robot Autom Lett 6(4):6458–6465

    Article  Google Scholar 

  19. Lee S, Lim H, Myung H (2022) Patchwork++: fast and robust ground segmentation solving partial under-segmentation using 3d point cloud. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 13276–13283

  20. Choi S, Ji G, Park J, Kim H, Mun J, Lee JH, Hwangbo J (2023) Learning quadrupedal locomotion on deformable terrain. Sci Robot 8(74):2256

    Article  Google Scholar 

  21. Kumar A, Fu Z, Pathak D, Malik J (2021) Rma: rapid motor adaptation for legged robots. Robotics: Science and Systems XVII

  22. Ji G, Mun J, Kim H, Hwangbo J (2022) Concurrent training of a control policy and a state estimator for dynamic and robust legged locomotion. IEEE Robot Autom Lett 7(2):4630–4637

    Article  Google Scholar 

  23. Nahrendra IMA, Yu B, Myung H (2023) Dreamwaq: learning robust quadrupedal locomotion with implicit terrain imagination via deep reinforcement learning. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), pp 5078–5084

  24. Luo Z, Dong Y, Li X, Huang R, Shu Z, Xiao E, Lu P (2024) Moral: learning morphologically adaptive locomotion controller for quadrupedal robots on challenging terrains. IEEE Robot Autom Lett 9(5):4019–4026

    Article  Google Scholar 

  25. Pinto L, Andrychowicz M, Welinder P, Zaremba W, Abbeel P (2018) Asymmetric actor critic for image-based robot learning. In: 14th Robotics: Science and Systems, RSS 2018

  26. Andrychowicz OM, Baker B, Chociej M, Jozefowicz R, McGrew B, Pachocki J, Petron A, Plappert M, Powell G, Ray A et al (2020) Learning dexterous in-hand manipulation. Int J Robot Res 39(1):3–20

    Article  Google Scholar 

  27. Rashid T, Samvelyan M, De Witt CS, Farquhar G, Foerster J, Whiteson S (2020) Monotonic value function factorisation for deep multi-agent reinforcement learning. J Mach Learn Res 21(178):1–51

    MathSciNet  Google Scholar 

  28. Wu J, Xin G, Qi C, Xue Y (2023) Learning robust and agile legged locomotion using adversarial motion priors. IEEE Robot Autom Lett

  29. Margolis GB, Yang G, Paigwar K, Chen T, Agrawal P (2024) Rapid locomotion via reinforcement learning. Int J Robot Res 43(4):572–587

    Article  Google Scholar 

  30. Peng XB, Abbeel P, Levine S, Panne M (2018) Deepmimic: example-guided deep reinforcement learning of physics-based character skills. ACM Trans Graph (TOG) 37(4):1–14

    Google Scholar 

  31. Shi H, Zhou B, Zeng H, Wang F, Dong Y, Li J, Wang K, Tian H, Meng MQ-H (2022) Reinforcement learning with evolutionary trajectory generator: a general approach for quadrupedal locomotion. IEEE Robot Autom Lett 7(2):3085–3092

    Article  Google Scholar 

  32. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

  33. Rudin N, Hoeller D, Reist P, Hutter M (2022) Learning to walk in minutes using massively parallel deep reinforcement learning. In: Conference on robot learning, pp 91–100

  34. Escontrela A, Peng XB, Yu W, Zhang T, Iscen A, Goldberg K, Abbeel P (2022) Adversarial motion priors make good substitutes for complex reward functions. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 25–32

  35. Hwangbo J, Lee J, Dosovitskiy A, Bellicoso D, Tsounis V, Koltun V, Hutter M (2019) Learning agile and dynamic motor skills for legged robots. Sci Robot 4(26):5872

    Article  Google Scholar 

  36. Fu Z, Kumar A, Malik J, Pathak D (2022) Minimizing energy consumption leads to the emergence of gaits in legged robots. In: Conference on robot learning, pp 928–937

  37. Xie Z, Ling HY, Kim NH, Panne M (2020) Allsteps: curriculum-driven learning of stepping stone skills. Comput Graphics Forum 39

  38. Muzio AF, Maximo MR, Yoneyama T (2022) Deep reinforcement learning for humanoid robot behaviors. J Intell Robot Syst 105(1):12

    Article  Google Scholar 

  39. Bonardi A, James S, Davison AJ (2020) Learning one-shot imitation from humans without humans. IEEE Robot Autom Lett 5(2):3533–3539

  40. Zhu W, Guo X, Owaki D, Kutsuzawa K, Hayashibe M (2021) A survey of sim-to-real transfer techniques applied to reinforcement learning for bioinspired robots. IEEE Trans Neural Netw Learn Syst 34(7):3444–3459

    Article  Google Scholar 

Download references

Acknowledgements

Funding from the National Nature Science Foundation of China and the Open Projects Program of the State Key Laboratory of Multimodal Artificial Intelligence Systems are gratefully acknowledged

Funding

This work was supported by the National Nature Science Foundation of China (62373016, 61873008) and the Open Projects Program of the State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS-2023-22).

Author information

Authors and Affiliations

  1. Beijing Key Laboratory of Computing Intelligence and Intelligent Systems, Beijing, 100124, China

    Guoyu Zuo, Yong Wang, Daoxiong Gong & Shuangyue Yu

  2. Faculty of Information Technology, Beijing University of Technology, Beijing, 100124, China

    Guoyu Zuo, Yong Wang, Daoxiong Gong & Shuangyue Yu

Authors
  1. Guoyu Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daoxiong Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shuangyue Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

In this work, Guoyu Zuo first proposed the research idea and approach of the paper and provided project guidance. The construction of the framework and the experimental platform was completed by Guoyu Zuo and Yong Wang. The code implementation, development, experiment testing, and data analysis were done by Yong Wang. The first draft of the manuscript was written by Yong Wang and Shuangyue Yu. Guoyu Zuo, Yong Wang, Daoxiong Gong, and Shuangyue Yu provided valuable suggestions and feedback on the draft. Guoyu Zuo, Daoxiong Gong, and Shuangyue Yu also made some important revisions to the final paper. All authors contributed to the article and approved the submitted version.

Corresponding author

Correspondence to Shuangyue Yu.

Ethics declarations

Ethics approval

The authors declare that they have no conflict of interest. This paper has not been previously published, it is published with the permission of the authors’ institution, and all authors of this paper are responsible for the authenticity of the data in the paper.

Consent to participate

All authors of this paper have been informed of the revision and publication of the paper, have checked all data, figures, and tables in the manuscript, and are responsible for their truthfulness and accuracy. Names of all contributing authors: Guoyu Zuo, Yong Wang, Shuangyue Yu.

Consent for publication

The publication has been approved by all co-authors.

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zuo, G., Wang, Y., Gong, D. et al. Learning quadrupedal locomotion on tough terrain using an asymmetric terrain feature mining network. Appl Intell 54, 11547–11563 (2024). https://doi.org/10.1007/s10489-024-05782-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05782-7

Keywords

Search

Navigation