The Need for Speed: Pruning Transformers with One Recipe
Samir Khaki Konstantinos N. Plataniotis
We introduce the $\textbf{O}$ne-shot $\textbf{P}$runing $\textbf{T}$echnique
for $\textbf{I}$nterchangeable $\textbf{N}$etworks ($\textbf{OPTIN}$) framework
as a tool to increase the efficiency of pre-trained transformer architectures
$\textit{without requiring re-training}$. Recent works have explored improving
transformer efficiency, however often incur computationally expensive
re-training procedures or depend on architecture-specific characteristics, thus
impeding practical wide-scale adoption. To address these shortcomings, the
OPTIN framework leverages intermediate feature distillation, capturing the
long-range dependencies of model parameters (coined $\textit{trajectory}$), to
produce state-of-the-art results on natural language, image classification,
transfer learning, and semantic segmentation tasks $\textit{without
re-training}$. Given a FLOP constraint, the OPTIN framework will compress the
network while maintaining competitive accuracy performance and improved
throughput. Particularly, we show a $\leq 2$% accuracy degradation from NLP
baselines and a $0.5$% improvement from state-of-the-art methods on image
classification at competitive FLOPs reductions. We further demonstrate the
generalization of tasks and architecture with comparative performance using
Mask2Former for semantic segmentation and cnn-style networks. OPTIN presents
one of the first one-shot efficient frameworks for compressing transformer
architectures that generalizes well across different class domains, in
particular: natural language and image-related tasks, without
$\textit{re-training}$.