TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenes
Bu Jin Yupeng Zheng Pengfei Li Weize Li Yuhang Zheng Sujie Hu Xinyu Liu Jinwei Zhu Zhijie Yan Haiyang Sun Kun Zhan Peng Jia Xiaoxiao Long Yilun Chen Hao Zhao
3D dense captioning stands as a cornerstone in achieving a comprehensive
understanding of 3D scenes through natural language. It has recently witnessed
remarkable achievements, particularly in indoor settings. However, the
exploration of 3D dense captioning in outdoor scenes is hindered by two major
challenges: 1) the \textbf{domain gap} between indoor and outdoor scenes, such
as dynamics and sparse visual inputs, makes it difficult to directly adapt
existing indoor methods; 2) the \textbf{lack of data} with comprehensive
box-caption pair annotations specifically tailored for outdoor scenes. To this
end, we introduce the new task of outdoor 3D dense captioning. As input, we
assume a LiDAR point cloud and a set of RGB images captured by the panoramic
camera rig. The expected output is a set of object boxes with captions. To
tackle this task, we propose the TOD3Cap network, which leverages the BEV
representation to generate object box proposals and integrates Relation
Q-Former with LLaMA-Adapter to generate rich captions for these objects. We
also introduce the TOD3Cap dataset, the largest one to our knowledge for 3D
dense captioning in outdoor scenes, which contains 2.3M descriptions of 64.3K
outdoor objects from 850 scenes. Notably, our TOD3Cap network can effectively
localize and caption 3D objects in outdoor scenes, which outperforms baseline
methods by a significant margin (+9.6 CiDEr@0.5IoU). Code, data, and models are
publicly available at https://github.com/jxbbb/TOD3Cap.