Masked Image Modeling via Dynamic Token Morphing
Taekyung Kim Dongyoon Han Byeongho Heo
Masked Image Modeling (MIM) arises as a promising option for Vision
Transformers among various self-supervised learning (SSL) methods. The essence
of MIM lies in token-wise masked patch predictions, with targets patchified
from images; or generated by pre-trained tokenizers or models. We argue targets
from the pre-trained models usually exhibit spatial inconsistency, which makes
it excessively challenging for the model to follow to learn more discriminative
representations. To mitigate the issue, we introduce a novel self-supervision
signal based on Dynamic Token Morphing (DTM), which dynamically aggregates
contextually related tokens. DTM can be generally applied to various SSL
frameworks, yet we propose a simple MIM that employs DTM to effectively improve
the performance barely introducing extra training costs. Our experiments on
ImageNet-1K and ADE20K evidently demonstrate the superiority of our methods.
Furthermore, the comparative evaluation of iNaturalist and Fine-grained Visual
Classification datasets further validates the transferability of our method on
various downstream tasks. Our code will be released publicly.