ORacle: Large Vision-Language Models for Knowledge-Guided Holistic OR
  Domain Modeling
Ege Özsoy Chantal Pellegrini Matthias Keicher Nassir Navab
Every day, countless surgeries are performed worldwide, each within the
distinct settings of operating rooms (ORs) that vary not only in their setups
but also in the personnel, tools, and equipment used. This inherent diversity
poses a substantial challenge for achieving a holistic understanding of the OR,
as it requires models to generalize beyond their initial training datasets. To
reduce this gap, we introduce ORacle, an advanced vision-language model
designed for holistic OR domain modeling, which incorporates multi-view and
temporal capabilities and can leverage external knowledge during inference,
enabling it to adapt to previously unseen surgical scenarios. This capability
is further enhanced by our novel data augmentation framework, which
significantly diversifies the training dataset, ensuring ORacle's proficiency
in applying the provided knowledge effectively. In rigorous testing, in scene
graph generation, and downstream tasks on the 4D-OR dataset, ORacle not only
demonstrates state-of-the-art performance but does so requiring less data than
existing models. Furthermore, its adaptability is displayed through its ability
to interpret unseen views, actions, and appearances of tools and equipment.
This demonstrates ORacle's potential to significantly enhance the scalability
and affordability of OR domain modeling and opens a pathway for future
advancements in surgical data science. We will release our code and data upon
acceptance.