Adapter Pruning using Tropical Characterization
Rishabh Bhardwaj Tushar Vaidya Soujanya Poria
Adapters are widely popular parameter-efficient transfer learning approaches
in natural language processing that insert trainable modules in between layers
of a pre-trained language model. Apart from several heuristics, however, there
has been a lack of studies analyzing the optimal number of adapter parameters
needed for downstream applications. In this paper, we propose an adapter
pruning approach by studying the tropical characteristics of trainable modules.
We cast it as an optimization problem that aims to prune parameters from the
adapter layers without changing the orientation of underlying tropical
hypersurfaces. Our experiments on five NLP datasets show that tropical geometry
tends to identify more relevant parameters to prune when compared with the
magnitude-based baseline, while a combined approach works best across the
tasks.