MuHRL: Hierarchical reinforcement learning control framework for a 7-DoF humanoid musculoskeletal robotic arm

Abstract

Humanoid robots, featuring human-like appearances and learning capabilities, can be seamlessly integrated into various human-robot interaction scenarios. This work presents a lightweight pneumatic musculoskeletal robotic arm featuring a seven-degrees-of-freedom joint arrangement mimicking that of a human arm. Its compact joint design and antagonistic muscle configuration simplify control complexity while maintaining a human-like appearance and movement capability. To enable efficient learning of manipulation tasks, Musculoskeletal Hierarchical Reinforcement Learning (MuHRL) is introduced. MuHRL is designed based on the mechanical structure of the proposed robotic arm and employs a two-level hierarchical policy that decouples top-level task planning from bottom-level joint angle control. The top level incorporates an incremental target posture to balance response speed and precise control, optimizing task performance, while the bottom level adopts a reusable multi-network angle control policy to improve angle control precision. Simulation experiments demonstrate that MuHRL exhibits superior sample efficiency and task performance than standard reinforcement learning and hierarchical reinforcement learning methods across three benchmark tasks. A force control experiment conducted on the hardware platform further validates its effectiveness on the musculoskeletal robot.