HiFi: High-Information Attention Heads Hold for Parameter-Efficient
  Model Adaptation
Anchun Gui Han Xiao
To fully leverage the advantages of large-scale pre-trained language models
(PLMs) on downstream tasks, it has become a ubiquitous adaptation paradigm to
fine-tune the entire parameters of PLMs. However, this paradigm poses issues of
inefficient updating and resource over-consuming for fine-tuning in data-scarce
and resource-limited scenarios, because of the large scale of parameters in
PLMs. To alleviate these concerns, in this paper, we propose a
parameter-efficient fine-tuning method HiFi, that is, only the highly
informative and strongly correlated attention heads for the specific task are
fine-tuned. To search for those significant attention heads, we develop a novel
framework to analyze the effectiveness of heads. Specifically, we first model
the relationship between heads into a graph from two perspectives of
information richness and correlation, and then apply PageRank algorithm to
determine the relative importance of each head. Extensive experiments on the
GLUE benchmark demonstrate the effectiveness of our method, and show that HiFi
obtains state-of-the-art performance over the prior baselines.