Physical Perception Network and an All-weather Multi-modality Benchmark
  for Adverse Weather Image Fusion
Xilai Li Wuyang Liu Xiaosong Li Haishu Tan
Multi-modality image fusion (MMIF) integrates the complementary information
from different modal images to provide comprehensive and objective
interpretation of a scenes. However, existing MMIF methods lack the ability to
resist different weather interferences in real-life scenarios, preventing them
from being useful in practical applications such as autonomous driving. To
bridge this research gap, we proposed an all-weather MMIF model. Regarding deep
learning architectures, their network designs are often viewed as a black box,
which limits their multitasking capabilities. For deweathering module, we
propose a physically-aware clear feature prediction module based on an
atmospheric scattering model that can deduce variations in light transmittance
from both scene illumination and depth. For fusion module, We utilize a
learnable low-rank representation model to decompose images into low-rank and
sparse components. This highly interpretable feature separation allows us to
better observe and understand images. Furthermore, we have established a
benchmark for MMIF research under extreme weather conditions. It encompasses
multiple scenes under three types of weather: rain, haze, and snow, with each
weather condition further subdivided into various impact levels. Extensive
fusion experiments under adverse weather demonstrate that the proposed
algorithm has excellent detail recovery and multi-modality feature extraction
capabilities.