Mono3DVG: 3D Visual Grounding in Monocular Images
Yang Zhan Yuan Yuan Zhitong Xiong
We introduce a novel task of 3D visual grounding in monocular RGB images
using language descriptions with both appearance and geometry information.
Specifically, we build a large-scale dataset, Mono3DRefer, which contains 3D
object targets with their corresponding geometric text descriptions, generated
by ChatGPT and refined manually. To foster this task, we propose Mono3DVG-TR,
an end-to-end transformer-based network, which takes advantage of both the
appearance and geometry information in text embeddings for multi-modal learning
and 3D object localization. Depth predictor is designed to explicitly learn
geometry features. The dual text-guided adapter is proposed to refine
multiscale visual and geometry features of the referred object. Based on
depth-text-visual stacking attention, the decoder fuses object-level geometric
cues and visual appearance into a learnable query. Comprehensive benchmarks and
some insightful analyses are provided for Mono3DVG. Extensive comparisons and
ablation studies show that our method significantly outperforms all baselines.
The dataset and code will be publicly available at:
https://github.com/ZhanYang-nwpu/Mono3DVG.