MemorySeg: Online LiDAR Semantic Segmentation with a Latent Memory
Enxu Li Sergio Casas Raquel Urtasun
Semantic segmentation of LiDAR point clouds has been widely studied in recent
years, with most existing methods focusing on tackling this task using a single
scan of the environment. However, leveraging the temporal stream of
observations can provide very rich contextual information on regions of the
scene with poor visibility (e.g., occlusions) or sparse observations (e.g., at
long range), and can help reduce redundant computation frame after frame. In
this paper, we tackle the challenge of exploiting the information from the past
frames to improve the predictions of the current frame in an online fashion. To
address this challenge, we propose a novel framework for semantic segmentation
of a temporal sequence of LiDAR point clouds that utilizes a memory network to
store, update and retrieve past information. Our framework also includes a
regularizer that penalizes prediction variations in the neighborhood of the
point cloud. Prior works have attempted to incorporate memory in range view
representations for semantic segmentation, but these methods fail to handle
occlusions and the range view representation of the scene changes drastically
as agents nearby move. Our proposed framework overcomes these limitations by
building a sparse 3D latent representation of the surroundings. We evaluate our
method on SemanticKITTI, nuScenes, and PandaSet. Our experiments demonstrate
the effectiveness of the proposed framework compared to the state-of-the-art.