Propagate & Distill: Towards Effective Graph Learners Using
  Propagation-Embracing MLPs
Yong-Min Shin Won-Yong Shin
Recent studies attempted to utilize multilayer perceptrons (MLPs) to solve
semisupervised node classification on graphs, by training a student MLP by
knowledge distillation from a teacher graph neural network (GNN). While
previous studies have focused mostly on training the student MLP by matching
the output probability distributions between the teacher and student models
during distillation, it has not been systematically studied how to inject the
structural information in an explicit and interpretable manner. Inspired by
GNNs that separate feature transformation $T$ and propagation $\Pi$, we
re-frame the distillation process as making the student MLP learn both $T$ and
$\Pi$. Although this can be achieved by applying the inverse propagation
$\Pi^{-1}$ before distillation from the teacher, it still comes with a high
computational cost from large matrix multiplications during training. To solve
this problem, we propose Propagate & Distill (P&D), which propagates the output
of the teacher before distillation, which can be interpreted as an approximate
process of the inverse propagation. We demonstrate that P&D can readily improve
the performance of the student MLP.