SOccDPT: Semi-Supervised 3D Semantic Occupancy from Dense Prediction
  Transformers trained under memory constraints
Aditya Nalgunda Ganesh
We present SOccDPT, a memory-efficient approach for 3D semantic occupancy
prediction from monocular image input using dense prediction transformers. To
address the limitations of existing methods trained on structured traffic
datasets, we train our model on unstructured datasets including the Indian
Driving Dataset and Bengaluru Driving Dataset. Our semi-supervised training
pipeline allows SOccDPT to learn from datasets with limited labels by reducing
the requirement for manual labelling by substituting it with pseudo-ground
truth labels to produce our Bengaluru Semantic Occupancy Dataset. This broader
training enhances our model's ability to handle unstructured traffic scenarios
effectively. To overcome memory limitations during training, we introduce
patch-wise training where we select a subset of parameters to train each epoch,
reducing memory usage during auto-grad graph construction. In the context of
unstructured traffic and memory-constrained training and inference, SOccDPT
outperforms existing disparity estimation approaches as shown by the RMSE score
of 9.1473, achieves a semantic segmentation IoU score of 46.02% and operates at
a competitive frequency of 69.47 Hz. We make our code and semantic occupancy
dataset public.