Efficient Adapter Tuning of Pre-trained Speech Models for Automatic
  Speaker Verification
Mufan Sang John H. L. Hansen
With excellent generalization ability, self-supervised speech models have
shown impressive performance on various downstream speech tasks in the
pre-training and fine-tuning paradigm. However, as the growing size of
pre-trained models, fine-tuning becomes practically unfeasible due to heavy
computation and storage overhead, as well as the risk of overfitting. Adapters
are lightweight modules inserted into pre-trained models to facilitate
parameter-efficient adaptation. In this paper, we propose an effective adapter
framework designed for adapting self-supervised speech models to the speaker
verification task. With a parallel adapter design, our proposed framework
inserts two types of adapters into the pre-trained model, allowing the
adaptation of latent features within intermediate Transformer layers and output
embeddings from all Transformer layers. We conduct comprehensive experiments to
validate the efficiency and effectiveness of the proposed framework.
Experimental results on the VoxCeleb1 dataset demonstrate that the proposed
adapters surpass fine-tuning and other parameter-efficient transfer learning
methods, achieving superior performance while updating only 5% of the
parameters.