Teaching MLP More Graph Information: A Three-stage Multitask Knowledge
  Distillation Framework
Junxian Li Bin Shi Erfei Cui Hua Wei Qinghua Zheng
We study the challenging problem for inference tasks on large-scale graph
datasets of Graph Neural Networks: huge time and memory consumption, and try to
overcome it by reducing reliance on graph structure. Even though distilling
graph knowledge to student MLP is an excellent idea, it faces two major
problems of positional information loss and low generalization. To solve the
problems, we propose a new three-stage multitask distillation framework. In
detail, we use Positional Encoding to capture positional information. Also, we
introduce Neural Heat Kernels responsible for graph data processing in GNN and
utilize hidden layer outputs matching for better performance of student MLP's
hidden layers. To the best of our knowledge, it is the first work to include
hidden layer distillation for student MLP on graphs and to combine graph
Positional Encoding with MLP. We test its performance and robustness with
several settings and draw the conclusion that our work can outperform well with
good stability.