OccFlowNet: Towards Self-supervised Occupancy Estimation via
  Differentiable Rendering and Occupancy Flow
Simon Boeder Fabian Gigengack Benjamin Risse
Semantic occupancy has recently gained significant traction as a prominent 3D
scene representation. However, most existing methods rely on large and costly
datasets with fine-grained 3D voxel labels for training, which limits their
practicality and scalability, increasing the need for self-monitored learning
in this domain. In this work, we present a novel approach to occupancy
estimation inspired by neural radiance field (NeRF) using only 2D labels, which
are considerably easier to acquire. In particular, we employ differentiable
volumetric rendering to predict depth and semantic maps and train a 3D network
based on 2D supervision only. To enhance geometric accuracy and increase the
supervisory signal, we introduce temporal rendering of adjacent time steps.
Additionally, we introduce occupancy flow as a mechanism to handle dynamic
objects in the scene and ensure their temporal consistency. Through extensive
experimentation we demonstrate that 2D supervision only is sufficient to
achieve state-of-the-art performance compared to methods using 3D labels, while
outperforming concurrent 2D approaches. When combining 2D supervision with 3D
labels, temporal rendering and occupancy flow we outperform all previous
occupancy estimation models significantly. We conclude that the proposed
rendering supervision and occupancy flow advances occupancy estimation and
further bridges the gap towards self-supervised learning in this domain.