Compressing Vision Transformers for Low-Resource Visual Learning
Eric Youn Sai Mitheran J Sanjana Prabhu Siyuan Chen
Vision transformer (ViT) and its variants have swept through visual learning
leaderboards and offer state-of-the-art accuracy in tasks such as image
classification, object detection, and semantic segmentation by attending to
different parts of the visual input and capturing long-range spatial
dependencies. However, these models are large and computation-heavy. For
instance, the recently proposed ViT-B model has 86M parameters making it
impractical for deployment on resource-constrained devices. As a result, their
deployment on mobile and edge scenarios is limited. In our work, we aim to take
a step toward bringing vision transformers to the edge by utilizing popular
model compression techniques such as distillation, pruning, and quantization.
  Our chosen application environment is an unmanned aerial vehicle (UAV) that
is battery-powered and memory-constrained, carrying a single-board computer on
the scale of an NVIDIA Jetson Nano with 4GB of RAM. On the other hand, the UAV
requires high accuracy close to that of state-of-the-art ViTs to ensure safe
object avoidance in autonomous navigation, or correct localization of humans in
search-and-rescue. Inference latency should also be minimized given the
application requirements. Hence, our target is to enable rapid inference of a
vision transformer on an NVIDIA Jetson Nano (4GB) with minimal accuracy loss.
This allows us to deploy ViTs on resource-constrained devices, opening up new
possibilities in surveillance, environmental monitoring, etc. Our
implementation is made available at https://github.com/chensy7/efficient-vit.