Creating a Digital Twin of Spinal Surgery: A Proof of Concept
Jonas Hein Frederic Giraud Lilian Calvet Alexander Schwarz Nicola Alessandro Cavalcanti Sergey Prokudin Mazda Farshad Siyu Tang Marc Pollefeys Fabio Carrillo Philipp Fürnstahl
Surgery digitalization is the process of creating a virtual replica of
real-world surgery, also referred to as a surgical digital twin (SDT). It has
significant applications in various fields such as education and training,
surgical planning, and automation of surgical tasks. Given their detailed
representations of surgical procedures, SDTs are an ideal foundation for
machine learning methods, enabling automatic generation of training data. In
robotic surgery, SDTs can provide realistic virtual environments in which
robots may learn through trial and error. In this paper, we present a proof of
concept (PoC) for surgery digitalization that is applied to an ex-vivo spinal
surgery performed in realistic conditions. The proposed digitalization focuses
on the acquisition and modelling of the geometry and appearance of the entire
surgical scene. We employ five RGB-D cameras for dynamic 3D reconstruction of
the surgeon, a high-end camera for 3D reconstruction of the anatomy, an
infrared stereo camera for surgical instrument tracking, and a laser scanner
for 3D reconstruction of the operating room and data fusion. We justify the
proposed methodology, discuss the challenges faced and further extensions of
our prototype. While our PoC partially relies on manual data curation, its high
quality and great potential motivate the development of automated methods for
the creation of SDTs. The quality of our SDT can be assessed in a rendered
video available at https://youtu.be/LqVaWGgaTMY .