WeatherProof: A Paired-Dataset Approach to Semantic Segmentation in
  Adverse Weather
Blake Gella Howard Zhang Rishi Upadhyay Tiffany Chang Matthew Waliman Yunhao Ba Alex Wong Achuta Kadambi
The introduction of large, foundational models to computer vision has led to
drastically improved performance on the task of semantic segmentation. However,
these existing methods exhibit a large performance drop when testing on images
degraded by weather conditions such as rain, fog, or snow. We introduce a
general paired-training method that can be applied to all current foundational
model architectures that leads to improved performance on images in adverse
weather conditions. To this end, we create the WeatherProof Dataset, the first
semantic segmentation dataset with accurate clear and adverse weather image
pairs, which not only enables our new training paradigm, but also improves the
evaluation of the performance gap between clear and degraded segmentation. We
find that training on these paired clear and adverse weather frames which share
an underlying scene results in improved performance on adverse weather data.
With this knowledge, we propose a training pipeline which accentuates the
advantages of paired-data training using consistency losses and language
guidance, which leads to performance improvements by up to 18.4% as compared to
standard training procedures.