Dynamic Association Learning of Self-Attention and Convolution in Image
  Restoration
Kui Jiang Xuemei Jia Wenxin Huang Wenbin Wang Zheng Wang Junjun Jiang
CNNs and Self attention have achieved great success in multimedia
applications for dynamic association learning of self-attention and convolution
in image restoration. However, CNNs have at least two shortcomings: 1) limited
receptive field; 2) static weight of sliding window at inference, unable to
cope with the content diversity.In view of the advantages and disadvantages of
CNNs and Self attention, this paper proposes an association learning method to
utilize the advantages and suppress their shortcomings, so as to achieve
high-quality and efficient inpainting. We regard rain distribution reflects the
degradation location and degree, in addition to the rain distribution
prediction. Thus, we propose to refine background textures with the predicted
degradation prior in an association learning manner. As a result, we accomplish
image deraining by associating rain streak removal and background recovery,
where an image deraining network and a background recovery network are designed
for two subtasks. The key part of association learning is a novel multi-input
attention module. It generates the degradation prior and produces the
degradation mask according to the predicted rainy distribution. Benefited from
the global correlation calculation of SA, MAM can extract the informative
complementary components from the rainy input with the degradation mask, and
then help accurate texture restoration. Meanwhile, SA tends to aggregate
feature maps with self-attention importance, but convolution diversifies them
to focus on the local textures. A hybrid fusion network involves one residual
Transformer branch and one encoder-decoder branch. The former takes a few
learnable tokens as input and stacks multi-head attention and feed-forward
networks to encode global features of the image. The latter, conversely,
leverages the multi-scale encoder-decoder to represent contexture knowledge.