EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards
  Embodied AI
Tai Wang Xiaohan Mao Chenming Zhu Runsen Xu Ruiyuan Lyu Peisen Li Xiao Chen Wenwei Zhang Kai Chen Tianfan Xue Xihui Liu Cewu Lu Dahua Lin Jiangmiao Pang
In the realm of computer vision and robotics, embodied agents are expected to
explore their environment and carry out human instructions. This necessitates
the ability to fully understand 3D scenes given their first-person observations
and contextualize them into language for interaction. However, traditional
research focuses more on scene-level input and output setups from a global
view. To address the gap, we introduce EmbodiedScan, a multi-modal, ego-centric
3D perception dataset and benchmark for holistic 3D scene understanding. It
encompasses over 5k scans encapsulating 1M ego-centric RGB-D views, 1M language
prompts, 160k 3D-oriented boxes spanning over 760 categories, some of which
partially align with LVIS, and dense semantic occupancy with 80 common
categories. Building upon this database, we introduce a baseline framework
named Embodied Perceptron. It is capable of processing an arbitrary number of
multi-modal inputs and demonstrates remarkable 3D perception capabilities, both
within the two series of benchmarks we set up, i.e., fundamental 3D perception
tasks and language-grounded tasks, and in the wild. Codes, datasets, and
benchmarks will be available at https://github.com/OpenRobotLab/EmbodiedScan.