Generalizability of Mixture of Domain-Specific Adapters from the Lens of
  Signed Weight Directions and its Application to Effective Model Pruning
Tuc Nguyen Thai Le
Several parameter-efficient fine-tuning methods based on adapters have been
proposed as a streamlined approach to incorporate not only a single specialized
knowledge into existing Pre-Trained Language Models (PLMs) but also multiple of
them at once. Recent works such as AdapterSoup propose to mix not all but only
a selective sub-set of domain-specific adapters during inference via model
weight averaging to optimize performance on novel, unseen domains with
excellent computational efficiency. However, the essential generalizability of
this emerging weight-space adapter mixing mechanism on unseen, in-domain
examples remains unexplored. Thus, in this study, we conduct a comprehensive
analysis to elucidate the generalizability of domain-specific adapter mixtures
in in-domain evaluation. We also provide investigations into the inner workings
of the mixture of domain-specific adapters by analyzing their weight signs,
yielding critical analysis on the negative correlation between their fraction
of weight sign difference and their mixtures' generalizability. All source code
will be published.