Picking the Underused Heads: A Network Pruning Perspective of Attention
  Head Selection for Fusing Dialogue Coreference Information
Zhengyuan Liu Nancy F. Chen
The Transformer-based models with the multi-head self-attention mechanism are
widely used in natural language processing, and provide state-of-the-art
results. While the pre-trained language backbones are shown to implicitly
capture certain linguistic knowledge, explicitly incorporating structure-aware
features can bring about further improvement on the downstream tasks. However,
such enhancement often requires additional neural components and increases
training parameter size. In this work, we investigate the attention head
selection and manipulation strategy for feature injection from a network
pruning perspective, and conduct a case study on dialogue summarization. We
first rank attention heads in a Transformer-based summarizer with layer-wise
importance. We then select the underused heads through extensive analysis, and
inject structure-aware features by manipulating the selected heads.
Experimental results show that the importance-based head selection is effective
for feature injection, and dialogue summarization can be improved by
incorporating coreference information via head manipulation.