Unveiling the Unseen Potential of Graph Learning through MLPs: Effective
  Graph Learners Using Propagation-Embracing MLPs
Yong-Min Shin Won-Yong Shin
Recent studies attempted to utilize multilayer perceptrons (MLPs) to solve
semi-supervised node classification on graphs, by training a student MLP by
knowledge distillation (KD) from a teacher graph neural network (GNN). While
previous studies have focused mostly on training the student MLP by matching
the output probability distributions between the teacher and student models
during KD, it has not been systematically studied how to inject the structural
information in an explicit and interpretable manner. Inspired by GNNs that
separate feature transformation $T$ and propagation $\Pi$, we re-frame the KD
process as enabling the student MLP to explicitly learn both $T$ and $\Pi$.
Although this can be achieved by applying the inverse propagation $\Pi^{-1}$
before distillation from the teacher GNN, it still comes with a high
computational cost from large matrix multiplications during training. To solve
this problem, we propose Propagate & Distill (P&D), which propagates the output
of the teacher GNN before KD and can be interpreted as an approximate process
of the inverse propagation $\Pi^{-1}$. Through comprehensive evaluations using
real-world benchmark datasets, we demonstrate the effectiveness of P&D by
showing further performance boost of the student MLP.