Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion
  Approach for 3D VQA
Wentao Mo Yang Liu
In 3D Visual Question Answering (3D VQA), the scarcity of fully annotated
data and limited visual content diversity hampers the generalization to novel
scenes and 3D concepts (e.g., only around 800 scenes are utilized in ScanQA and
SQA dataset). Current approaches resort supplement 3D reasoning with 2D
information. However, these methods face challenges: either they use top-down
2D views that introduce overly complex and sometimes question-irrelevant visual
clues, or they rely on globally aggregated scene/image-level representations
from 2D VLMs, losing the fine-grained vision-language correlations. To overcome
these limitations, our approach utilizes question-conditional 2D view selection
procedure, pinpointing semantically relevant 2D inputs for crucial visual
clues. We then integrate this 2D knowledge into the 3D-VQA system via a
two-branch Transformer structure. This structure, featuring a Twin-Transformer
design, compactly combines 2D and 3D modalities and captures fine-grained
correlations between modalities, allowing them mutually augmenting each other.
Integrating proposed mechanisms above, we present BridgeQA, that offers a fresh
perspective on multi-modal transformer-based architectures for 3D-VQA.
Experiments validate that BridgeQA achieves state-of-the-art on 3D-VQA datasets
and significantly outperforms existing solutions. Code is available at
$\href{https://github.com/matthewdm0816/BridgeQA}{\text{this URL}}$.