Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery
  Tickets from Large Models
Ajay Jaiswal Shiwei Liu Tianlong Chen Ying Ding Zhangyang Wang
Large pre-trained transformers have been receiving explosive attention in the
past few years, due to their wide adaptability for numerous downstream
applications via fine-tuning, but their exponentially increasing parameter
counts are becoming a primary hurdle to even just fine-tune them without
industry-standard hardware. Recently, Lottery Ticket Hypothesis (LTH) and its
variants, have been exploited to prune these large pre-trained models
generating subnetworks that can achieve similar performance as their dense
counterparts, but LTH pragmatism is enormously inhibited by repetitive full
training and pruning routine of iterative magnitude pruning (IMP) which worsens
with increasing model size. Motivated by the recent observations of model
soups, which suggest that fine-tuned weights of multiple models can be merged
to a better minima, we propose Instant Soup Pruning (ISP) to generate lottery
ticket quality subnetworks, using a fraction of the original IMP cost by
replacing the expensive intermediate pruning stages of IMP with computationally
efficient weak mask generation and aggregation routine. More specifically,
during the mask generation stage, ISP takes a small handful of iterations using
varying training protocols and data subsets to generate many weak and noisy
subnetworks, and superpose them to average out the noise creating a
high-quality denoised subnetwork. Our extensive experiments and ablation on two
popular large-scale pre-trained models: CLIP (unexplored in pruning till date)
and BERT across multiple benchmark vision and language datasets validate the
effectiveness of ISP compared to several state-of-the-art pruning methods.
Codes are available at: \url{https://github.com/VITA-Group/instant_soup}