General-purpose foundation models for increased autonomy in
  robot-assisted surgery
Samuel Schmidgall Ji Woong Kim Alan Kuntz Ahmed Ezzat Ghazi Axel Krieger
The dominant paradigm for end-to-end robot learning focuses on optimizing
task-specific objectives that solve a single robotic problem such as picking up
an object or reaching a target position. However, recent work on high-capacity
models in robotics has shown promise toward being trained on large collections
of diverse and task-agnostic datasets of video demonstrations. These models
have shown impressive levels of generalization to unseen circumstances,
especially as the amount of data and the model complexity scale. Surgical robot
systems that learn from data have struggled to advance as quickly as other
fields of robot learning for a few reasons: (1) there is a lack of existing
large-scale open-source data to train models, (2) it is challenging to model
the soft-body deformations that these robots work with during surgery because
simulation cannot match the physical and visual complexity of biological
tissue, and (3) surgical robots risk harming patients when tested in clinical
trials and require more extensive safety measures. This perspective article
aims to provide a path toward increasing robot autonomy in robot-assisted
surgery through the development of a multi-modal, multi-task,
vision-language-action model for surgical robots. Ultimately, we argue that
surgical robots are uniquely positioned to benefit from general-purpose models
and provide three guiding actions toward increased autonomy in robot-assisted
surgery.