Longer-range Contextualized Masked Autoencoder
Taekyung Kim Sanghyuk Chun Byeongho Heo Dongyoon Han
Masked image modeling (MIM) has emerged as a promising self-supervised
learning (SSL) strategy. The MIM pre-training facilitates learning powerful
representations using an encoder-decoder framework by randomly masking some
input pixels and reconstructing the masked pixels from the remaining ones.
However, as the encoder is trained with partial pixels, the MIM pre-training
can suffer from a low capability of understanding long-range dependency. This
limitation may hinder its capability to fully understand multiple-range
dependencies, resulting in narrow highlighted regions in the attention map that
may incur accuracy drops. To mitigate the limitation, We propose a
self-supervised learning framework, named Longer-range Contextualized Masked
Autoencoder (LC-MAE). LC-MAE effectively leverages a global context
understanding of visual representations while simultaneously reducing the
spatial redundancy of input at the same time. Our method steers the encoder to
learn from entire pixels in multiple views while also learning local
representation from sparse pixels. As a result, LC-MAE learns more
discriminative representations, leading to a performance improvement of
achieving 84.2% top-1 accuracy with ViT-B on ImageNet-1K with 0.6%p gain. We
attribute the success to the enhanced pre-training method, as evidenced by the
singular value spectrum and attention analyses. Finally, LC-MAE achieves
significant performance gains at the downstream semantic segmentation and
fine-grained visual classification tasks; and on diverse robust evaluation
metrics. Our code will be publicly available.