Neural Fields for 3D Tracking of Anatomy and Surgical Instruments in
  Monocular Laparoscopic Video Clips
Beerend G. A. Gerats Jelmer M. Wolterink Seb P. Mol Ivo A. M. J. Broeders
Laparoscopic video tracking primarily focuses on two target types: surgical
instruments and anatomy. The former could be used for skill assessment, while
the latter is necessary for the projection of virtual overlays. Where
instrument and anatomy tracking have often been considered two separate
problems, in this paper, we propose a method for joint tracking of all
structures simultaneously. Based on a single 2D monocular video clip, we train
a neural field to represent a continuous spatiotemporal scene, used to create
3D tracks of all surfaces visible in at least one frame. Due to the small size
of instruments, they generally cover a small part of the image only, resulting
in decreased tracking accuracy. Therefore, we propose enhanced class weighting
to improve the instrument tracks. We evaluate tracking on video clips from
laparoscopic cholecystectomies, where we find mean tracking accuracies of 92.4%
for anatomical structures and 87.4% for instruments. Additionally, we assess
the quality of depth maps obtained from the method's scene reconstructions. We
show that these pseudo-depths have comparable quality to a state-of-the-art
pre-trained depth estimator. On laparoscopic videos in the SCARED dataset, the
method predicts depth with an MAE of 2.9 mm and a relative error of 9.2%. These
results show the feasibility of using neural fields for monocular 3D
reconstruction of laparoscopic scenes.