A Comprehensive Evaluation of Parameter-Efficient Fine-Tuning on
  Software Engineering Tasks
Wentao Zou Qi Li Jidong Ge Chuanyi Li Xiaoyu Shen Liguo Huang Bin Luo
Pre-trained models (PTMs) have achieved great success in various Software
Engineering (SE) downstream tasks following the ``pre-train then fine-tune''
paradigm. As fully fine-tuning all parameters of PTMs can be computationally
expensive, a widely used solution is parameter-efficient fine-tuning (PEFT),
which freezes PTMs while introducing extra parameters. Though work has been
done to test PEFT methods in the SE field, a comprehensive evaluation is still
lacking. This paper aims to fill in this gap by evaluating the effectiveness of
five PEFT methods on eight PTMs and four SE downstream tasks. For different
tasks and PEFT methods, we seek answers to the following research questions: 1)
Is it more effective to use PTMs trained specifically on source code, or is it
sufficient to use PTMs trained on natural language text? 2) What is the impact
of varying model sizes? 3) How does the model architecture affect the
performance? Besides effectiveness, we also discuss the efficiency of PEFT
methods, concerning the costs of required training time and GPU resource
consumption. We hope that our findings can provide a deeper understanding of
PEFT methods on various PTMs and SE downstream tasks. All the codes and data
are available at \url{https://github.com/zwtnju/PEFT.git}.