Attention is all you need for boosting graph convolutional neural
  network
Yinwei Wu
Graph Convolutional Neural Networks (GCNs) possess strong capabilities for
processing graph data in non-grid domains. They can capture the topological
logical structure and node features in graphs and integrate them into nodes'
final representations. GCNs have been extensively studied in various fields,
such as recommendation systems, social networks, and protein molecular
structures. With the increasing application of graph neural networks, research
has focused on improving their performance while compressing their size. In
this work, a plug-in module named Graph Knowledge Enhancement and Distillation
Module (GKEDM) is proposed. GKEDM can enhance node representations and improve
the performance of GCNs by extracting and aggregating graph information via
multi-head attention mechanism. Furthermore, GKEDM can serve as an auxiliary
transferor for knowledge distillation. With a specially designed attention
distillation method, GKEDM can distill the knowledge of large teacher models
into high-performance and compact student models. Experiments on multiple
datasets demonstrate that GKEDM can significantly improve the performance of
various GCNs with minimal overhead. Furthermore, it can efficiently transfer
distilled knowledge from large teacher networks to small student networks via
attention distillation.