Hierarchical Attention Models for Multi-Relational Graphs
Roshni G. Iyer Wei Wang Yizhou Sun
We present Bi-Level Attention-Based Relational Graph Convolutional Networks
(BR-GCN), unique neural network architectures that utilize masked
self-attentional layers with relational graph convolutions, to effectively
operate on highly multi-relational data. BR-GCN models use bi-level attention
to learn node embeddings through (1) node-level attention, and (2)
relation-level attention. The node-level self-attentional layers use
intra-relational graph interactions to learn relation-specific node embeddings
using a weighted aggregation of neighborhood features in a sparse subgraph
region. The relation-level self-attentional layers use inter-relational graph
interactions to learn the final node embeddings using a weighted aggregation of
relation-specific node embeddings. The BR-GCN bi-level attention mechanism
extends Transformer-based multiplicative attention from the natural language
processing (NLP) domain, and Graph Attention Networks (GAT)-based attention, to
large-scale heterogeneous graphs (HGs). On node classification, BR-GCN
outperforms baselines from 0.29% to 14.95% as a stand-alone model, and on link
prediction, BR-GCN outperforms baselines from 0.02% to 7.40% as an auto-encoder
model. We also conduct ablation studies to evaluate the quality of BR-GCN's
relation-level attention and discuss how its learning of graph structure may be
transferred to enrich other graph neural networks (GNNs). Through various
experiments, we show that BR-GCN's attention mechanism is both scalable and
more effective in learning compared to state-of-the-art GNNs.