Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Experiment-free exoskeleton assistance via learning in simulation

Nature volume 630, pages 353–359 (2024)Cite this article

Subjects

Abstract

Exoskeletons have enormous potential to improve human locomotive performance1,2,3. However, their development and broad dissemination are limited by the requirement for lengthy human tests and handcrafted control laws2. Here we show an experiment-free method to learn a versatile control policy in simulation. Our learning-in-simulation framework leverages dynamics-aware musculoskeletal and exoskeleton models and data-driven reinforcement learning to bridge the gap between simulation and reality without human experiments. The learned controller is deployed on a custom hip exoskeleton that automatically generates assistance across different activities with reduced metabolic rates by 24.3%, 13.1% and 15.4% for walking, running and stair climbing, respectively. Our framework may offer a generalizable and scalable strategy for the rapid development and widespread adoption of a variety of assistive robots for both able-bodied and mobility-impaired individuals.

Access through your institution
Buy or subscribe

This is a preview of subscription content, access via your institution

Access options

Access through your institution

Buy this article

  • Purchase on SpringerLink
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Experiment-free optimization of exoskeleton assistance through learning in simulation.
Fig. 2: Learning-in-simulation framework.
Fig. 3: Generalizable and adaptive assistive torque by the learned controller.
Fig. 4: Representative assistive torque during various activities and locomotion transitions.
Fig. 5: Reduction in metabolic rate during walking, running and stair climbing.

Similar content being viewed by others

Data availability

All data supporting the findings of this study are available in the Article and its Supplementary Information.  Source data are provided with this paper.

Code availability

Pseudocode for the learning-in-simulation algorithm and training process can be found in the GitHub repository https://github.com/IntelligentRobotLearning/pseudocode_learning_in_simulation.

References

  1. Siviy, C. et al. Opportunities and challenges in the development of exoskeletons for locomotor assistance. Nat. Biomed. Eng. 7, 456–472 (2023).

    Article  PubMed  Google Scholar 

  2. Slade, P., Kochenderfer, M. J., Delp, S. L. & Collins, S. H. Personalizing exoskeleton assistance while walking in the real world. Nature 610, 277–282 (2022).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  3. Awad, L. N. et al. A soft robotic exosuit improves walking in patients after stroke. Sci. Transl. Med. 9, eaai9084 (2017).

    Article  PubMed  Google Scholar 

  4. Geyer, H., Seyfarth, A. & Blickhan, R. Compliant leg behaviour explains basic dynamics of walking and running. Proc. R. Soc. B 273, 2861–2867 (2006).

    Article  PubMed  PubMed Central  Google Scholar 

  5. Zhang, J. et al. Human-in-the-loop optimization of exoskeleton assistance during walking. Science 356, 1280–1284 (2017).

    Article  ADS  CAS  PubMed  Google Scholar 

  6. Ferris, D., Sawicki, G. & Domingo, A. Powered lower limb orthoses for gait rehabilitation. Top. Spinal Cord Inj. Rehabil. 11, 34–49 (2005).

    Article  PubMed  Google Scholar 

  7. Veneman, J. F. et al. Design and evaluation of the LOPES exoskeleton robot for interactive gait rehabilitation. IEEE Trans. Neural Syst. Rehabil. Eng. 15, 379–386 (2007).

    Article  PubMed  Google Scholar 

  8. Ding, Y., Kim, M., Kuindersma, S. & Walsh, C. J. Human-in-the-loop optimization of hip assistance with a soft exosuit during walking. Sci. Robot. 3, eaar5438 (2018).

    Article  PubMed  Google Scholar 

  9. Koller, J. R., Jacobs, D. A., Ferris, D. P. & Remy, C. D. Learning to walk with an adaptive gain proportional myoelectric controller for a robotic ankle exoskeleton. J. Neuroeng. Rehabil. 12, 97 (2015).

  10. Zhang, Q. et al. Imposing healthy hip motion pattern and range by exoskeleton control for individualized assistance. IEEE Robot. Autom. Lett. 7, 11126–11133 (2022).

    Article  Google Scholar 

  11. Dembia, C. L., Bianco, N. A., Falisse, A., Hicks, J. L. & Delp, S. L. Opensim moco: musculoskeletal optimal control. PLoS Comput. Biol. 16, e1008493 (2020).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  12. Song, S. et al. Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation. J. Neuroeng. Rehabil. 18, 126 (2021).

  13. Gordon, D. F., McGreavy, C., Christou, A. & Vijayakumar, S. Human-in-the-loop optimization of exoskeleton assistance via online simulation of metabolic cost. IEEE Trans. Robot. 38, 1410–1429 (2022).

    Article  Google Scholar 

  14. Durandau, G., Rampeltshammer, W. F., van der Kooij, H. & Sartori, M. Neuromechanical model-based adaptive control of bilateral ankle exoskeletons: biological joint torque and electromyogram reduction across walking conditions. IEEE Trans. Robot. 38, 1380–1394 (2022).

    Article  Google Scholar 

  15. Lim, B. et al. Delayed output feedback control for gait assistance with a robotic hip exoskeleton. IEEE Trans. Robot. 35, 1055–1062 (2019).

    Article  Google Scholar 

  16. Meuleman, J., Van Asseldonk, E., Van Oort, G., Rietman, H. & Van Der Kooij, H. LOPES II—design and evaluation of an admittance controlled gait training robot with shadow-leg approach. IEEE Trans. Neural Syst. Rehabil. Eng. 24, 352–363 (2015).

    Article  PubMed  Google Scholar 

  17. Molinaro, D. D., Kang, I., Camargo, J., Gombolay, M. C. & Young, A. J. Subject-independent, biological hip moment estimation during multimodal overground ambulation using deep learning. IEEE Trans. Med. Robot. Bionics 4, 219–229 (2022).

    Article  Google Scholar 

  18. Shepherd, M. K., Molinaro, D. D., Sawicki, G. S. & Young, A. J. Deep learning enables exoboot control to augment variable-speed walking. IEEE Robot. Autom. Lett. 7, 3571–3577 (2022).

    Article  Google Scholar 

  19. Bellegarda, G. & Ijspeert, A. CPG-RL: learning central pattern generators for quadruped locomotion. IEEE Robot. Autom. Lett. 7, 12547–12554 (2022).

    Article  Google Scholar 

  20. Li, Z. et al. Reinforcement learning for robust parameterized locomotion control of bipedal robots. In 2021 IEEE International Conference on Robotics and Automation (eds Meng, Q.-H. & Sun, Y.) 2811–2817 (IEEE, 2021).

  21. Siekmann, J., Godse, Y., Fern, A. & Hurst, J. Sim-to-real learning of all common bipedal gaits via periodic reward composition. In IEEE International Conference on Robotics and Automation (eds Meng, Q.-H. & Sun, Y.) 7309–7315 (IEEE, 2021).

  22. Wen, Y., Si, J., Brandt, A., Gao, X. & Huang, H. H. Online reinforcement learning control for the personalization of a robotic knee prosthesis. IEEE Trans. Cybernet. 50, 2346–2356 (2019).

    Article  Google Scholar 

  23. Luo, S. et al. Reinforcement learning and control of a lower extremity exoskeleton for squat assistance. Front. Robot. AI 8, 702845 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  24. Luo, S. et al. Robust walking control of a lower limb rehabilitation exoskeleton coupled with a musculoskeletal model via deep reinforcement learning. J. Neuroeng. Rehabil. 20, 34 (2023).

  25. CMU Graphics Lab Motion Capture Database. http://mocap.cs.cmu.edu/ (Carnegie Mellon University; accessed 10 November 2021).

  26. Adadi, A. A survey on data‐efficient algorithms in big data era. J. Big Data 8, 24 (2021).

    Article  Google Scholar 

  27. Salvato, E., Fenu, G., Medvet, E. & Pellegrino, F. A. Crossing the reality gap: a survey on sim-to-real transferability of robot controllers in reinforcement learning. IEEE Access 9, 153171–153187 (2021).

    Article  Google Scholar 

  28. Kim, J. et al. Autonomous and portable soft exosuit for hip extension assistance with online walking and running detection algorithm. In 2018 IEEE International Conference on Robotics and Automation (eds Zelinksy, A. & Park, F.) 5473–5480 (IEEE, 2018).

  29. Zhang, X. et al. Enhancing gait assistance control robustness of a hip exosuit by means of machine learning. IEEE Robot. Autom. Lett. 7, 7566–7573 (2022).

    Article  Google Scholar 

  30. Cao, W., Chen, C., Hu, H., Fang, K. & Wu, X. Effect of hip assistance modes on metabolic cost of walking with a soft exoskeleton. IEEE Trans. Autom. Sci. Eng. 18, 426–436 (2020).

    Article  Google Scholar 

  31. Kim, J. et al. Reducing the energy cost of walking with low assistance levels through optimized hip flexion assistance from a soft exosuit. Sci. Rep. 12, 11004 (2022).

    Article  ADS  CAS  PubMed  PubMed Central  Google Scholar 

  32. Seo, K., Lee, J. & Park, Y. J. Autonomous hip exoskeleton saves metabolic cost of walking uphill. In International Conference on Rehabilitation Robotics (ed. Masia, L.) 246–251 (IEEE, 2017).

  33. Nasiri, R., Ahmadi, A. & Ahmadabadi, M. N. Reducing the energy cost of human running using an unpowered exoskeleton. IEEE Trans. Neural Syst. Rehabil. Eng. 26, 2026–2032 (2018).

    Article  PubMed  Google Scholar 

  34. Ning, C., Li, Y., Feng, K., Gong, Z. & Zhang, T. Soochow Exo: a lightweight hip exoskeleton driven by series elastic actuator with active-type continuously variable transmission. In IEEE International Conference on Mechatronics and Automation (eds Hirata, H., Xu, W. & Guo, J.) 1432–1437 (IEEE, 2022).

  35. Kim, D. S. et al. A wearable hip-assist robot reduces the cardiopulmonary metabolic energy expenditure during stair ascent in elderly adults: a pilot cross-sectional study. BMC Geriatr. 18, 230 (2018).

  36. Kim, J. et al. Reducing the metabolic rate of walking and running with a versatile, portable exosuit. Science 365, 668–672 (2019).

    Article  ADS  CAS  PubMed  Google Scholar 

  37. Sawicki, G. S., Beck, O. N., Kang, I. & Young, A. J. The exoskeleton expansion: improving walking and running economy. J. Neuroeng. Rehabil. 17, 25 (2020).

  38. Koos, S., Mouret, J.-B. & Doncieux, S. The transferability approach: crossing the reality gap in evolutionary robotics. IEEE Trans. Evol. Comput. 17, 122–145 (2012).

    Article  Google Scholar 

  39. Halilaj, E. et al. Machine learning in human movement biomechanics: best practices, common pitfalls and new opportunities. J. Biomech. 81, 1–11 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  40. Lee, G. et al. Reducing the metabolic cost of running with a tethered soft exosuit. Sci. Robot. 2, eaan6708 (2017).

    Article  PubMed  Google Scholar 

  41. Lee, D., Kwak, E. C., McLain, B. J., Kang, I. & Young, A. J. Effects of assistance during early stance phase using a robotic knee orthosis on energetics, muscle activity and joint mechanics during incline and decline walking. IEEE Trans. Neural Syst. Rehabil. Eng. 28, 914–923 (2020).

    Article  PubMed  Google Scholar 

  42. Franks, P. W. et al. Comparing optimized exoskeleton assistance of the hip, knee and ankle in single and multi-joint configurations. Wearable Technol. 2, e16 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  43. Nuckols, R. W. et al. Individualization of exosuit assistance based on measured muscle dynamics during versatile walking. Sci. Robot. 6, eabj1362 (2021).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  44. Shepertycky, M., Burton, S., Dickson, A., Liu, Y.-F. & Li, Q. Removing energy with an exoskeleton reduces the metabolic cost of walking. Science 372, 957–960 (2021).

    Article  CAS  PubMed  Google Scholar 

  45. Mooney, L. M. & Herr, H. M. Biomechanical walking mechanisms underlying the metabolic reduction caused by an autonomous exoskeleton. J. Neuroeng. Rehabil. 13, 4 (2016).

  46. Seo, K., Lee, J., Lee, Y., Ha, T. & Shim, Y. Fully autonomous hip exoskeleton saves metabolic cost of walking. In 2016 IEEE International Conference on Robotics and Automation (eds Kragic, D., Bicchi, A. & De Luca, A.) 4628–4635 (IEEE, 2016).

  47. Seth, A. et al. OpenSim: simulating musculoskeletal dynamics and neuromuscular control to study human and animal movement. PLoS Comput. Biol. 14, e1006223 (2018).

    Article  PubMed  PubMed Central  Google Scholar 

  48. Blake, O. M. & Wakeling, J. M. Estimating changes in metabolic power from EMG. Springerplus 2, 229 (2013).

  49. Bruns, R. E., Vos, P. & Wedge, R. D. Electromyography as a surrogate for estimating metabolic energy expenditure during locomotion. Med. Eng. Phys. 109, 103899 (2022).

    Article  PubMed  Google Scholar 

  50. Silder, A., Besier, T. & Delp, S. L. Predicting the metabolic cost of incline walking from muscle activity and walking mechanics. J. Biomech. 45, 1842–1849 (2012).

    Article  PubMed  PubMed Central  Google Scholar 

  51. Jackson, R. W. & Collins, S. H. An experimental comparison of the relative benefits of work and torque assistance in ankle exoskeletons. J. Appl. Physiol. 119, 541–557 (2015).

    Article  PubMed  Google Scholar 

  52. Hortobágyi, T., Finch, A., Solnik, S., Rider, P. & DeVita, P. Association between muscle activation and metabolic cost of walking in young and old adults. J. Gerontol. A 66, 541–547 (2011).

    Article  Google Scholar 

  53. Lee, S., Park, M., Lee, K. & Lee, J. Scalable muscle-actuated human simulation and control. ACM Trans. Graph. 38, 73 (2019).

  54. Thelen, D. G. Adjustment of muscle mechanics model parameters to simulate dynamic contractions in older adults. J. Biomech. Eng. 125, 70–77 (2003).

    Article  PubMed  Google Scholar 

  55. Zhou, X. & Zheng, L. Model-based comparison of passive and active assistance designs in an occupational upper limb exoskeleton for overhead lifting. IISE Trans. Occup. Ergon. Hum. Factors 9, 167–185 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  56. Hwangbo, J. et al. Learning agile and dynamic motor skills for legged robots. Sci. Robot. 4, eaau5872 (2019).

    Article  PubMed  Google Scholar 

  57. Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V. & Hutter, M. Learning quadrupedal locomotion over challenging terrain. Sci. Robot. 5, eabc5986 (2020).

    Article  PubMed  Google Scholar 

  58. Tan, J. et al. Sim-to-real: learning agile locomotion for quadruped robots. Preprint at https://doi.org/10.48550/arXiv.1804.10332 (2018).

  59. Tobin, J. et al. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (eds Zhang, H. & Vaughan, R.) 23–30 (IEEE, 2017).

  60. Yu, S. et al. Quasi-direct drive actuation for a lightweight hip exoskeleton with high backdrivability and high bandwidth. IEEE ASME Trans. Mechatron. 25, 1794–1802 (2020).

  61. Huang, T.-H. et al. Modeling and stiffness-based continuous torque control of lightweight quasi-direct-drive knee exoskeletons for versatile walking assistance. IEEE Trans. Robot. 38, 1442–1459 (2022).

Download references

Acknowledgements

We thank Y. K. Chen at the Massachusetts Institute of Technology for constructive feedback and discussion of this work. This work was supported in part by the National Science Foundation (NSF) CAREER award CMMI 1944655, National Institute on Disability, Independent Living and Rehabilitation Research (NIDILRR) DRRP 90DPGE0019, Switzer Research Distinguished Fellow (SFGE22000372), NSF Future of Work 2231419 and National Institute of Health (NIH) 1R01EB035404.

Author information

Authors and Affiliations

  1. Lab of Biomechatronics and Intelligent Robotics, Department of Mechanical and Aerospace Engineering, North Carolina State University, Raleigh, NC, USA

    Shuzhen Luo, Menghan Jiang, Sainan Zhang, Junxi Zhu, Shuangyue Yu, Israel Dominguez Silva, Tian Wang & Hao Su

  2. Department of Mechanical Engineering, Embry-Riddle Aeronautical University, Daytona Beach, FL, USA

    Shuzhen Luo

  3. Neurobionics Lab, Department of Robotics, Department of Mechanical Engineering, University of Michigan, Ann Arbor, MI, USA

    Elliott Rouse

  4. Department of Computer Science, University of California, Los Angeles, CA, USA

    Bolei Zhou

  5. Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea

    Hyunwoo Yuk

  6. Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ, USA

    Xianlian Zhou

  7. Joint NCSU/UNC Department of Biomedical Engineering, North Carolina State University, Raleigh, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Hao Su

Authors
  1. Shuzhen Luo
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Menghan Jiang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Sainan Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Junxi Zhu
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Shuangyue Yu
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Israel Dominguez Silva
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Tian Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Elliott Rouse
    View author publications

    You can also search for this author inPubMed Google Scholar

  9. Bolei Zhou
    View author publications

    You can also search for this author inPubMed Google Scholar

  10. Hyunwoo Yuk
    View author publications

    You can also search for this author inPubMed Google Scholar

  11. Xianlian Zhou
    View author publications

    You can also search for this author inPubMed Google Scholar

  12. Hao Su
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

H.S. first proposed the research idea and approach of the paper and provided the project guidance with input from E.J.R., B.Z. and X.Z. S.L., M.J. and H.S. were responsible for the design of this experiment. The methodology, code implementation and development were conducted by S.L. on the basis of a framework created by S.L. and X.Z. The mechatronics design and fabrication of the exoskeleton were done by I.D.S., M.J., S.Z. and J.Z. The experiments and data analysis were conducted by S.L., M.J., S.Z. and J.Z. The first draft of the manuscript was prepared by S.L., M.J., J.Z., S.Y., I.D.S., T.W. and H.S. Important revisions were made by S.L., J.Z., T.W., E.J.R., B.Z., H.Y., X.Z. and H.S. to the final paper. All authors contributed to the article and approved the submitted version.

Corresponding author

Correspondence to Hao Su.

Ethics declarations

Competing interests

S.L. and H.S. are co-inventors on intellectual property related to the controller discussed in this work. H.S. is a co-founder of and has a financial interest in Picasso Intelligence, LLC. The terms of this arrangement have been reviewed and approved by NC State University in accordance with its policy on objectivity in research. The remaining authors declare no competing interests.

Peer review

Peer review information

Nature thanks Joonho Lee and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended Data Fig. 1 Overview of the learning-in-simulation control framework.

a,b, Schematic illustrations for the learning-in-simulation architecture (a) and the control structure for online deployment (b).

Extended Data Fig. 2 Motion imitation neural network for versatile activities.

Schematic illustrations for the autonomous learning framework of the reference motions (walking, running, stair climbing) from datasets based on human kinematics input and joint torque command output.

Extended Data Fig. 3 Muscle coordination neural network.

Schematic illustrations for the muscle coordination neural network based on human joint torque input and human muscle actuation output.

Extended Data Fig. 4 Exoskeleton control neural network.

Schematic illustrations for the exoskeleton control neural network based on exoskeleton state history input and joint torque command output.

Extended Data Fig. 5 Exoskeleton design.

a–c, Overall view of the whole system (a), actuator (b) and electronics (c).

Extended Data Fig. 6 Experiment protocol for metabolic rate and kinematic data collection during walking, running and stair climbing (also available on Protocol Exchange at: https://doi.org/10.21203/rs.3.pex-2632/v1)

.

Extended Data Table 1 Participant information
Full size table
Extended Data Table 2 Summary of experiments
Full size table

Supplementary information

Supplementary Information

Supplementary Text 1–7, Methods, Figs. 1–7, Tables 1–8 and references.

Supplementary Video 1

We present a control approach that learns assistive control strategies in simulation and can be transferred to a physical wearable robot to generate continuous assistance for several locomotion activities, including walking, running and stair climbing.

Supplementary Video 2

Human response to robots is slow and controller development typically requires 24–60 min human testing and is limited to single activity (mostly walking) control only. Our method leverages dynamics-aware and data-driven simulation. It requires no human testing and can be immediately deployed to a physical exoskeleton for several activities. We only need simulation once for 8 h to learn the assistive control policy that is transferred to microcontrollers for real-time control of physical exoskeletons.

Supplementary Video 3

The learned controller in simulation is transferred to a physical exoskeleton for real-time control that immediately improves mobility. Using control policies trained in simulation, the controller is versatile to assist several locomotion modes and leads to significant metabolic expenditure savings by 24.3%, 13.1% and 15.4% during walking, running and stair climbing, respectively, compared with no exoskeleton conditions.

Supplementary Video 4

The robot learns control strategies by simultaneous training of muscle-coordination neural network and robot controller neural network. It also learns multilocomotion control by an activity imitation neural network. We bridge the sim-to-real gap by domain randomization of muscle models and robot parameters and the control policy only requires one wearable sensor per leg to control the exoskeleton.

Supplementary Video 5

The robot donning takes about 2 min and doffing takes less than 1 min. The 3.2 kg low-profile hip exoskeleton does not affect the natural range of motion and thus can assist various movements for heterogeneous able-bodied individuals.

Source data

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Luo, S., Jiang, M., Zhang, S. et al. Experiment-free exoskeleton assistance via learning in simulation. Nature 630, 353–359 (2024). https://doi.org/10.1038/s41586-024-07382-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41586-024-07382-4

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics