Input Compression with Positional Consistency for Efficient Training and
  Inference of Transformer Neural Networks
Amrit Nagarajan Anand Raghunathan
Transformers have rapidly increased in popularity in recent years, achieving
state-of-the-art performance in processing text, images, audio and video.
However, Transformers present large computational requirements for both
training and inference, and are prone to overfitting during training. To
address these challenges, we present Input Compression with Positional
Consistency (ICPC), a new data augmentation method that, unlike prior
augmentation techniques, simultaneously improves both generalization and
training efficiency. ICPC applies varying levels of compression to each
training sample in each epoch. This leads to smaller input sequences being
processed by the Transformer, and hence faster training, while also alleviating
overfitting by presenting each input with different compression levels. We
introduce a consistency-aware position selection method in ICPC that enables
accurate processing of compressed inputs without any changes to the underlying
Transformer architecture. We detail compression-based augmentation methods for
four different modalities -- insignificant word pruning for text, resolution
modulation for images, spatio-temporal resolution modulation for videos, and
spectogram size modulation for audio. ICPC also enables efficient
variable-effort inference, where samples are first inferred at high compression
levels, and progressively re-evaluated with lower compression for more
challenging inputs. On 9 diverse tasks spanning 4 different modalities, ICPC
improves accuracy by up to 1%, while also accelerating training and inference
by up to 2.9X and 2.6X, respectively. Code is available at
https://github.com/amrnag/ICPC.