Memory-based Adapters for Online 3D Scene Perception
Xiuwei Xu Chong Xia Ziwei Wang Linqing Zhao Yueqi Duan Jie Zhou Jiwen Lu
In this paper, we propose a new framework for online 3D scene perception.
Conventional 3D scene perception methods are offline, i.e., take an already
reconstructed 3D scene geometry as input, which is not applicable in robotic
applications where the input data is streaming RGB-D videos rather than a
complete 3D scene reconstructed from pre-collected RGB-D videos. To deal with
online 3D scene perception tasks where data collection and perception should be
performed simultaneously, the model should be able to process 3D scenes frame
by frame and make use of the temporal information. To this end, we propose an
adapter-based plug-and-play module for the backbone of 3D scene perception
model, which constructs memory to cache and aggregate the extracted RGB-D
features to empower offline models with temporal learning ability.
Specifically, we propose a queued memory mechanism to cache the supporting
point cloud and image features. Then we devise aggregation modules which
directly perform on the memory and pass temporal information to current frame.
We further propose 3D-to-2D adapter to enhance image features with strong
global context. Our adapters can be easily inserted into mainstream offline
architectures of different tasks and significantly boost their performance on
online tasks. Extensive experiments on ScanNet and SceneNN datasets demonstrate
our approach achieves leading performance on three 3D scene perception tasks
compared with state-of-the-art online methods by simply finetuning existing
offline models, without any model and task-specific designs.
\href{https://xuxw98.github.io/Online3D/}{Project page}.