Local Masking Meets Progressive Freezing: Crafting Efficient Vision
  Transformers for Self-Supervised Learning
Utku Mert Topcuoglu Erdem Akagündüz
In this paper, we present an innovative approach to self-supervised learning
for Vision Transformers (ViTs), integrating local masked image modeling with
progressive layer freezing. This method focuses on enhancing the efficiency and
speed of initial layer training in ViTs. By systematically freezing specific
layers at strategic points during training, we reduce computational demands
while maintaining or improving learning capabilities. Our approach employs a
novel multi-scale reconstruction process that fosters efficient learning in
initial layers and enhances semantic comprehension across scales. The results
demonstrate a substantial reduction in training time (~12.5\%) with a minimal
impact on model accuracy (decrease in top-1 accuracy by 0.6\%). Our method
achieves top-1 and top-5 accuracies of 82.6\% and 96.2\%, respectively,
underscoring its potential in scenarios where computational resources and time
are critical. This work marks an advancement in the field of self-supervised
learning for computer vision. The implementation of our approach is available
at our project's GitHub repository: github.com/utkutpcgl/ViTFreeze.