HINT: High-quality INPainting Transformer with Mask-Aware Encoding and
  Enhanced Attention
Shuang Chen Amir Atapour-Abarghouei Hubert P. H. Shum
Existing image inpainting methods leverage convolution-based downsampling
approaches to reduce spatial dimensions. This may result in information loss
from corrupted images where the available information is inherently sparse,
especially for the scenario of large missing regions. Recent advances in
self-attention mechanisms within transformers have led to significant
improvements in many computer vision tasks including inpainting. However,
limited by the computational costs, existing methods cannot fully exploit the
efficacy of long-range modelling capabilities of such models. In this paper, we
propose an end-to-end High-quality INpainting Transformer, abbreviated as HINT,
which consists of a novel mask-aware pixel-shuffle downsampling module (MPD) to
preserve the visible information extracted from the corrupted image while
maintaining the integrity of the information available for high-level
inferences made within the model. Moreover, we propose a Spatially-activated
Channel Attention Layer (SCAL), an efficient self-attention mechanism
interpreting spatial awareness to model the corrupted image at multiple scales.
To further enhance the effectiveness of SCAL, motivated by recent advanced in
speech recognition, we introduce a sandwich structure that places feed-forward
networks before and after the SCAL module. We demonstrate the superior
performance of HINT compared to contemporary state-of-the-art models on four
datasets, CelebA, CelebA-HQ, Places2, and Dunhuang.