Benchmarking the Robustness of Panoptic Segmentation for Automated
  Driving
Yiting Wang Haonan Zhao Daniel Gummadi Mehrdad Dianati Kurt Debattista Valentina Donzella
Precise situational awareness is required for the safe decision-making of
assisted and automated driving (AAD) functions. Panoptic segmentation is a
promising perception technique to identify and categorise objects, impending
hazards, and driveable space at a pixel level. While segmentation quality is
generally associated with the quality of the camera data, a comprehensive
understanding and modelling of this relationship are paramount for AAD system
designers. Motivated by such a need, this work proposes a unifying pipeline to
assess the robustness of panoptic segmentation models for AAD, correlating it
with traditional image quality. The first step of the proposed pipeline
involves generating degraded camera data that reflects real-world noise
factors. To this end, 19 noise factors have been identified and implemented
with 3 severity levels. Of these factors, this work proposes novel models for
unfavourable light and snow. After applying the degradation models, three
state-of-the-art CNN- and vision transformers (ViT)-based panoptic segmentation
networks are used to analyse their robustness. The variations of the
segmentation performance are then correlated to 8 selected image quality
metrics. This research reveals that: 1) certain specific noise factors produce
the highest impact on panoptic segmentation, i.e. droplets on lens and Gaussian
noise; 2) the ViT-based panoptic segmentation backbones show better robustness
to the considered noise factors; 3) some image quality metrics (i.e. LPIPS and
CW-SSIM) correlate strongly with panoptic segmentation performance and
therefore they can be used as predictive metrics for network performance.