Data-efficient Event Camera Pre-training via Disentangled Masked
  Modeling
Zhenpeng Huang Chao Li Hao Chen Yongjian Deng Yifeng Geng Limin Wang
In this paper, we present a new data-efficient voxel-based self-supervised
learning method for event cameras. Our pre-training overcomes the limitations
of previous methods, which either sacrifice temporal information by converting
event sequences into 2D images for utilizing pre-trained image models or
directly employ paired image data for knowledge distillation to enhance the
learning of event streams. In order to make our pre-training data-efficient, we
first design a semantic-uniform masking method to address the learning
imbalance caused by the varying reconstruction difficulties of different
regions in non-uniform data when using random masking. In addition, we ease the
traditional hybrid masked modeling process by explicitly decomposing it into
two branches, namely local spatio-temporal reconstruction and global semantic
reconstruction to encourage the encoder to capture local correlations and
global semantics, respectively. This decomposition allows our selfsupervised
learning method to converge faster with minimal pre-training data. Compared to
previous approaches, our self-supervised learning method does not rely on
paired RGB images, yet enables simultaneous exploration of spatial and temporal
cues in multiple scales. It exhibits excellent generalization performance and
demonstrates significant improvements across various tasks with fewer
parameters and lower computational costs.