Mansformer: Efficient Transformer of Mixed Attention for Image
  Deblurring and Beyond
Pin-Hung Kuo Jinshan Pan Shao-Yi Chien Ming-Hsuan Yang
Transformer has made an enormous success in natural language processing and
high-level vision over the past few years. However, the complexity of
self-attention is quadratic to the image size, which makes it infeasible for
high-resolution vision tasks. In this paper, we propose the Mansformer, a
Transformer of mixed attention that combines multiple self-attentions, gate,
and multi-layer perceptions (MLPs), to explore and employ more possibilities of
self-attention. Taking efficiency into account, we design four kinds of
self-attention, whose complexities are all linear. By elaborate adjustment of
the tensor shapes and dimensions for the dot product, we split the typical
self-attention of quadratic complexity into four operations of linear
complexity. To adaptively merge these different kinds of self-attention, we
take advantage of an architecture similar to Squeeze-and-Excitation Networks.
Furthermore, we make it to merge the two-staged Transformer design into one
stage by the proposed gated-dconv MLP. Image deblurring is our main target,
while extensive quantitative and qualitative evaluations show that this method
performs favorably against the state-of-the-art methods far more than simply
deblurring. The source codes and trained models will be made available to the
public.