With the rapid development of robotic and artificial intelligence technologies, the autonomous decision-making capability of endoscopic surgical robot has been significantly enhanced, accompanied by growing demands for automation in non-decision-making tasks. This study focuses on the path planning for complex tasks in surgical robot by proposing a hierarchical reinforcement learning (HRL) framework based on the option framework. Within this framework, the Semi-Markov Decision Process (SMDP) is extended into an augmented Markov Decision Process (MDP) to optimize termination conditions and facilitate long-horizon task training. To address the sparse reward problem, a hierarchical reward function is designed with intrinsic temporal rewards specifically implemented for high-level policies. In addition, a Type-Shared Option Policy (TSOP) is proposed to enhance training efficiency. Experimental results demonstrate that the proposed HRL framework effectively improves both the success rate and stability of path planning for surgical robot in the da Vinci Research Kit (dVRK) simulation environment.
AbdelaalAELiuJHongN, et al. (2021) Parallelism in autonomous robotic surgery. IEEE Robotics and Automation Letters6(2): 1824–1831.
2.
ChenQLiuYWangP, et al. (2022) An autonomous obstacle avoidance method for dual-arm surgical robot based on the improved artificial potential field method. In: Intelligent robotics and applications, Harbin, China, 1–3 August, pp. 496–508. Cham: Springer.
3.
ChiuZYRichterFFunkEK, et al. (2021) Bimanual regrasping for suture needles using reinforcement learning for rapid motion planning. In: 2021 IEEE international conference on robotics and automation (ICRA), Xi’an, China, 30 May–5 June, pp.7737–7743. New York: IEEE.
4.
DankelmanJ (2004) Surgical robots and other training tools in minimally invasive surgery. In: 2004 IEEE international conference on systems, man and cybernetics (IEEE Cat. No.04CH37583), vol. 3, The Hague, 10–13 October, pp. 2459–2464. New York: IEEE.
5.
De MariaMMeneghettiIMosilloL, et al. (2024) Versius robotic surgical system: Cases series of 18 robot-assisted radical prostatectomies. BJU International133(2): 197–205.
6.
HaoLLiuDDuS, et al. (2022) An improved path planning algorithm based on artificial potential field and primal-dual neural network for surgical robot. Computer Methods and Programs in Biomedicine227: 107202.
7.
HeYZhaoBQiX, et al. (2020) Automatic surgical field of view control in robot-assisted nasal surgery. IEEE Robotics and Automation Letters6(1): 247–254.
8.
JianXSongYLiuD, et al. (2025) Motion planning and control of active robot in orthopedics surgery by CDMP-based imitation learning and constrained optimization. IEEE Transactions on Automation Science and Engineering22: 12197–12212.
9.
KimJWZhaoTZSchmidgallS, et al. (2024) Surgical robot transformer (SRT): Imitation learning for surgical tasks. arXiv preprint arXiv: 2407.12998.
10.
LeonardSWuKLKimY, et al. (2014) Smart tissue anastomosis robot (STAR): A vision-guided robotics system for laparoscopic suturing. IEEE Transactions on Biomedical Engineering61(4): 1305–1317.
11.
LiuGQiXLiM, et al. (2025) An automatic cutting plane planning method based on multi-objective optimization for robot-assisted laminectomy surgery. IEEE Robotics and Automation Letters10(3): 2343–2350.
12.
LiuXWangGLiuZ, et al. (2024) Hierarchical reinforcement learning integrating with human knowledge for practical robot skill learning in complex multi-stage manipulation. IEEE Transactions on Automation Science and Engineering21(3): 3852–3862.
13.
LvCZhuMGuoX, et al. (2025) Hierarchical reinforcement learning method for long-horizon path planning of stratospheric airship. Aerospace Science and Technology160: 110075.
14.
NguyenNDNguyenTNahavandiS, et al. (2019) Manipulating soft tissues by deep reinforcement learning for autonomous robotic surgery. In: 2019 IEEE international systems conference (SysCon), Orlando, FL, 8–11 April, pp. 1–7. New York: IEEE.
15.
OuYTavakoliM (2023) Sim-to-real surgical robot learning and autonomous planning for internal tissue points manipulation using reinforcement learning. IEEE Robotics and Automation Letters8(5): 2502–2509.
SaeidiHLeHNOpfermannJD, et al. (2019) Autonomous laparoscopic robotic suturing with a novel actuated suturing tool and 3D endoscope. In: 2019 international conference on robotics and automation (ICRA), Montreal, QC, Canada, 20–24 May, pp.1541–1547. New York: IEEE.
18.
SaeidiHOpfermannJDKamM, et al. (2022) Autonomous robotic laparoscopic surgery for intestinal anastomosis. Science Robotics7(62): eabj2908.
19.
SalisburyJKGuthartGS (2000) The Intuitive™ telesurgery system: Overview and application. In: Proceedings 2000 ICRA, Millennium conference, IEEE international conference on robotics and automation, Symposia proceedings (Cat. No.00CH37065), vol. 1, San Francisco, CA, 24–28 April, pp. 618–621. New York: IEEE.
20.
SamalaviciusNDulskasAJanusonisV, et al. (2022) Robotic colorectal surgery using the Senhance® robotic system: A single center experience. Techniques in Coloproctology26(6): 437–442.
21.
ShahkooAAAbinAA (2023) Deep reinforcement learning in continuous action space for autonomous robotic surgery. International Journal of Computer Assisted Radiology and Surgery18(3): 423–431.
22.
ShiHHeZHwangKS (2025) Adaptive path planning for wafer second probing via an attention-based hierarchical reinforcement learning framework with shared memory. Information Sciences710: 122089.
23.
SuHHuYLiJ, et al. (2020) Improving motion planning for surgical robot with active constraints. In: 2020 IEEE/RSJ international conference on intelligent robots and systems (IROS), Las Vegas, NV, 24 October 2020–24 January 2021, pp. 3151–3156. New York: IEEE.
24.
TaylorRFundaJEldridgeB, et al. (1995) A telerobotic assistant for laparoscopic surgery. IEEE Engineering in Medicine and Biology Magazine14(3): 279–288.
25.
WangYFUeckerDRYulunW (1996) Choreographed scope manoeuvring in robotically-assisted laparoscopy with active vision guidance. In: Proceedings third IEEE workshop on applications of computer vision, WACV’, Sarasota, FL, 2–4 December, pp. 187–192. New York: IEEE.
26.
ZargarzadehSMirzaeiMOuY, et al. (2025) From decision to action in surgical autonomy: Multi-modal large language models for robot assisted blood suction. IEEE Robotics and Automation Letters10(3): 2598–2605.
27.
ZhouHLinX (2025) Intelligent redundant manipulation for long horizon operations with multiple goal-conditioned hierarchical learning. Advanced Robotics39(6): 291–304.
