Adaptive Multi-modal Fusion of Spatially Variant Kernel Refinement with
  Diffusion Model for Blind Image Super-Resolution
Junxiong Lin Yan Wang Zeng Tao Boyang Wang Qing Zhao Haorang Wang Xuan Tong Xinji Mai Yuxuan Lin Wei Song Jiawen Yu Shaoqi Yan Wenqiang Zhang
Pre-trained diffusion models utilized for image generation encapsulate a
substantial reservoir of a priori knowledge pertaining to intricate textures.
Harnessing the potential of leveraging this a priori knowledge in the context
of image super-resolution presents a compelling avenue. Nonetheless, prevailing
diffusion-based methodologies presently overlook the constraints imposed by
degradation information on the diffusion process. Furthermore, these methods
fail to consider the spatial variability inherent in the estimated blur kernel,
stemming from factors such as motion jitter and out-of-focus elements in
open-environment scenarios. This oversight results in a notable deviation of
the image super-resolution effect from fundamental realities. To address these
concerns, we introduce a framework known as Adaptive Multi-modal Fusion of
\textbf{S}patially Variant Kernel Refinement with Diffusion Model for Blind
Image \textbf{S}uper-\textbf{R}esolution (SSR). Within the SSR framework, we
propose a Spatially Variant Kernel Refinement (SVKR) module. SVKR estimates a
Depth-Informed Kernel, which takes the depth information into account and is
spatially variant. Additionally, SVKR enhance the accuracy of depth information
acquired from LR images, allowing for mutual enhancement between the depth map
and blur kernel estimates. Finally, we introduce the Adaptive Multi-Modal
Fusion (AMF) module to align the information from three modalities:
low-resolution images, depth maps, and blur kernels. This alignment can
constrain the diffusion model to generate more authentic SR results.
Quantitative and qualitative experiments affirm the superiority of our
approach, while ablation experiments corroborate the effectiveness of the
modules we have proposed.