Interactive Multi-Head Self-Attention with Linear Complexity
Hankyul Kang Ming-Hsuan Yang Jongbin Ryu
We propose an efficient interactive method for multi-head self-attention via
decomposition. For existing methods using multi-head self-attention, the
attention operation of each head is computed independently. However, we show
that the interactions between cross-heads of the attention matrix enhance the
information flow of the attention operation. Considering that the attention
matrix of each head can be seen as a feature of networks, it is beneficial to
establish connectivity between them to capture interactions better. However, a
straightforward approach to capture the interactions between the cross-heads is
computationally prohibitive as the complexity grows substantially with the high
dimension of an attention matrix. In this work, we propose an effective method
to decompose the attention operation into query- and key-less components. This
will result in a more manageable size for the attention matrix, specifically
for the cross-head interactions. Expensive experimental results show that the
proposed cross-head interaction approach performs favorably against existing
efficient attention methods and state-of-the-art backbone models.