Performance Evaluation of Swin Vision Transformer Model using Gradient
  Accumulation Optimization Technique
Sanad Aburass Osama Dorgham
Vision Transformers (ViTs) have emerged as a promising approach for visual
recognition tasks, revolutionizing the field by leveraging the power of
transformer-based architectures. Among the various ViT models, Swin
Transformers have gained considerable attention due to their hierarchical
design and ability to capture both local and global visual features
effectively. This paper evaluates the performance of Swin ViT model using
gradient accumulation optimization (GAO) technique. We investigate the impact
of gradient accumulation optimization technique on the model's accuracy and
training time. Our experiments show that applying the GAO technique leads to a
significant decrease in the accuracy of the Swin ViT model, compared to the
standard Swin Transformer model. Moreover, we detect a significant increase in
the training time of the Swin ViT model when GAO model is applied. These
findings suggest that applying the GAO technique may not be suitable for the
Swin ViT model, and concern should be undertaken when using GAO technique for
other transformer-based models.