Real-time 3D semantic occupancy prediction for autonomous vehicles using
  memory-efficient sparse convolution
Samuel Sze Lars Kunze
In autonomous vehicles, understanding the surrounding 3D environment of the
ego vehicle in real-time is essential. A compact way to represent scenes while
encoding geometric distances and semantic object information is via 3D semantic
occupancy maps. State of the art 3D mapping methods leverage transformers with
cross-attention mechanisms to elevate 2D vision-centric camera features into
the 3D domain. However, these methods encounter significant challenges in
real-time applications due to their high computational demands during
inference. This limitation is particularly problematic in autonomous vehicles,
where GPU resources must be shared with other tasks such as localization and
planning. In this paper, we introduce an approach that extracts features from
front-view 2D camera images and LiDAR scans, then employs a sparse convolution
network (Minkowski Engine), for 3D semantic occupancy prediction. Given that
outdoor scenes in autonomous driving scenarios are inherently sparse, the
utilization of sparse convolution is particularly apt. By jointly solving the
problems of 3D scene completion of sparse scenes and 3D semantic segmentation,
we provide a more efficient learning framework suitable for real-time
applications in autonomous vehicles. We also demonstrate competitive accuracy
on the nuScenes dataset.