Language-driven All-in-one Adverse Weather Removal
Hao Yang Liyuan Pan Yan Yang Wei Liang
All-in-one (AiO) frameworks restore various adverse weather degradations with
a single set of networks jointly. To handle various weather conditions, an AiO
framework is expected to adaptively learn weather-specific knowledge for
different degradations and shared knowledge for common patterns. However,
existing methods: 1) rely on extra supervision signals, which are usually
unknown in real-world applications; 2) employ fixed network structures, which
restrict the diversity of weather-specific knowledge. In this paper, we propose
a Language-driven Restoration framework (LDR) to alleviate the aforementioned
issues. First, we leverage the power of pre-trained vision-language (PVL)
models to enrich the diversity of weather-specific knowledge by reasoning about
the occurrence, type, and severity of degradation, generating description-based
degradation priors. Then, with the guidance of degradation prior, we sparsely
select restoration experts from a candidate list dynamically based on a
Mixture-of-Experts (MoE) structure. This enables us to adaptively learn the
weather-specific and shared knowledge to handle various weather conditions
(e.g., unknown or mixed weather). Experiments on extensive restoration
scenarios show our superior performance (see Fig. 1). The source code will be
made available.