Surgical-VQLA: Transformer with Gated Vision-Language Embedding for
  Visual Question Localized-Answering in Robotic Surgery
Long Bai Mobarakol Islam Lalithkumar Seenivasan Hongliang Ren
Despite the availability of computer-aided simulators and recorded videos of
surgical procedures, junior residents still heavily rely on experts to answer
their queries. However, expert surgeons are often overloaded with clinical and
academic workloads and limit their time in answering. For this purpose, we
develop a surgical question-answering system to facilitate robot-assisted
surgical scene and activity understanding from recorded videos. Most of the
existing VQA methods require an object detector and regions based feature
extractor to extract visual features and fuse them with the embedded text of
the question for answer generation. However, (1) surgical object detection
model is scarce due to smaller datasets and lack of bounding box annotation;
(2) current fusion strategy of heterogeneous modalities like text and image is
naive; (3) the localized answering is missing, which is crucial in complex
surgical scenarios. In this paper, we propose Visual Question
Localized-Answering in Robotic Surgery (Surgical-VQLA) to localize the specific
surgical area during the answer prediction. To deal with the fusion of the
heterogeneous modalities, we design gated vision-language embedding (GVLE) to
build input patches for the Language Vision Transformer (LViT) to predict the
answer. To get localization, we add the detection head in parallel with the
prediction head of the LViT. We also integrate GIoU loss to boost localization
performance by preserving the accuracy of the question-answering model. We
annotate two datasets of VQLA by utilizing publicly available surgical videos
from MICCAI challenges EndoVis-17 and 18. Our validation results suggest that
Surgical-VQLA can better understand the surgical scene and localize the
specific area related to the question-answering. GVLE presents an efficient
language-vision embedding technique by showing superior performance over the
existing benchmarks.