Exploring the Application of Large-scale Pre-trained Models on Adverse
  Weather Removal
Zhentao Tan Yue Wu Qiankun Liu Qi Chu Le Lu Jieping Ye Nenghai Yu
Image restoration under adverse weather conditions (e.g., rain, snow and
haze) is a fundamental computer vision problem and has important indications
for various downstream applications. Different from early methods that are
specially designed for specific type of weather, most recent works tend to
remove various adverse weather effects simultaneously through either spatial
feature representation learning or semantic information embedding. Inspired by
the various successful applications of large-scale pre-trained models (e.g,
CLIP), in this paper, we explore the potential benefits of them for this task
through both spatial feature representation learning and semantic information
embedding aspects: 1) for spatial feature representation learning, we design a
Spatially-Adaptive Residual (\textbf{SAR}) Encoder to extract degraded areas
adaptively. To facilitate its training, we propose a Soft Residual Distillation
(\textbf{CLIP-SRD}) strategy to transfer the spatial knowledge from CLIP
between clean and adverse weather images; 2) for semantic information
embedding, we propose a CLIP Weather Prior (\textbf{CWP}) embedding module to
make the network handle different weather conditions adaptively. This module
integrates the sample specific weather prior extracted by CLIP image encoder
together with the distribution specific information learned by a set of
parameters, and embeds them through a cross attention mechanism. Extensive
experiments demonstrate that our proposed method can achieve state-of-the-art
performance under different and challenging adverse weather conditions. Code
will be made available.