Guided Image Restoration via Simultaneous Feature and Image Guided
  Fusion
Xinyi Liu Qian Zhao Jie Liang Hui Zeng Deyu Meng Lei Zhang
Guided image restoration (GIR), such as guided depth map super-resolution and
pan-sharpening, aims to enhance a target image using guidance information from
another image of the same scene. Currently, joint image filtering-inspired deep
learning-based methods represent the state-of-the-art for GIR tasks. Those
methods either deal with GIR in an end-to-end way by elaborately designing
filtering-oriented deep neural network (DNN) modules, focusing on the
feature-level fusion of inputs; or explicitly making use of the traditional
joint filtering mechanism by parameterizing filtering coefficients with DNNs,
working on image-level fusion. The former ones are good at recovering
contextual information but tend to lose fine-grained details, while the latter
ones can better retain textual information but might lead to content
distortions. In this work, to inherit the advantages of both methodologies
while mitigating their limitations, we proposed a Simultaneous Feature and
Image Guided Fusion (SFIGF) network, that simultaneously considers feature and
image-level guided fusion following the guided filter (GF) mechanism. In the
feature domain, we connect the cross-attention (CA) with GF, and propose a
GF-inspired CA module for better feature-level fusion; in the image domain, we
fully explore the GF mechanism and design GF-like structure for better
image-level fusion. Since guided fusion is implemented in both feature and
image domains, the proposed SFIGF is expected to faithfully reconstruct both
contextual and textual information from sources and thus lead to better GIR
results. We apply SFIGF to 4 typical GIR tasks, and experimental results on
these tasks demonstrate its effectiveness and general availability.