Towards Deep Attention in Graph Neural Networks: Problems and Remedies
Soo Yong Lee Fanchen Bu Jaemin Yoo Kijung Shin
Graph neural networks (GNNs) learn the representation of graph-structured
data, and their expressiveness can be further enhanced by inferring node
relations for propagation. Attention-based GNNs infer neighbor importance to
manipulate the weight of its propagation. Despite their popularity, the
discussion on deep graph attention and its unique challenges has been limited.
In this work, we investigate some problematic phenomena related to deep graph
attention, including vulnerability to over-smoothed features and smooth
cumulative attention. Through theoretical and empirical analyses, we show that
various attention-based GNNs suffer from these problems. Motivated by our
findings, we propose AEROGNN, a novel GNN architecture designed for deep graph
attention. AERO-GNN provably mitigates the proposed problems of deep graph
attention, which is further empirically demonstrated with (a) its adaptive and
less smooth attention functions and (b) higher performance at deep layers (up
to 64). On 9 out of 12 node classification benchmarks, AERO-GNN outperforms the
baseline GNNs, highlighting the advantages of deep graph attention. Our code is
available at https://github.com/syleeheal/AERO-GNN.