Value-Informed Skill Chaining for Policy Learning of Long-Horizon Tasks
  with Surgical Robot
Tao Huang Kai Chen Wang Wei Jianan Li Yonghao Long Qi Dou
Reinforcement learning is still struggling with solving long-horizon surgical
robot tasks which involve multiple steps over an extended duration of time due
to the policy exploration challenge. Recent methods try to tackle this problem
by skill chaining, in which the long-horizon task is decomposed into multiple
subtasks for easing the exploration burden and subtask policies are temporally
connected to complete the whole long-horizon task. However, smoothly connecting
all subtask policies is difficult for surgical robot scenarios. Not all states
are equally suitable for connecting two adjacent subtasks. An undesired
terminate state of the previous subtask would make the current subtask policy
unstable and result in a failed execution. In this work, we introduce
value-informed skill chaining (ViSkill), a novel reinforcement learning
framework for long-horizon surgical robot tasks. The core idea is to
distinguish which terminal state is suitable for starting all the following
subtask policies. To achieve this target, we introduce a state value function
that estimates the expected success probability of the entire task given a
state. Based on this value function, a chaining policy is learned to instruct
subtask policies to terminate at the state with the highest value so that all
subsequent policies are more likely to be connected for accomplishing the task.
We demonstrate the effectiveness of our method on three complex surgical robot
tasks from SurRoL, a comprehensive surgical simulation platform, achieving high
task success rates and execution efficiency. Code is available at
$\href{https://github.com/med-air/ViSkill}{\text{https://github.com/med-air/ViSkill}}$.