Distilling Large Language Models for Text-Attributed Graph Learning
Bo Pan Zheng Zhang Yifei Zhang Yuntong Hu Liang Zhao
Text-Attributed Graphs (TAGs) are graphs of connected textual documents.
Graph models can efficiently learn TAGs, but their training heavily relies on
human-annotated labels, which are scarce or even unavailable in many
applications. Large language models (LLMs) have recently demonstrated
remarkable capabilities in few-shot and zero-shot TAG learning, but they suffer
from scalability, cost, and privacy issues. Therefore, in this work, we focus
on synergizing LLMs and graph models with their complementary strengths by
distilling the power of LLMs to a local graph model on TAG learning. To address
the inherent gaps between LLMs (generative models for texts) and graph models
(discriminative models for graphs), we propose first to let LLMs teach an
interpreter with rich textual rationale and then let a student model mimic the
interpreter's reasoning without LLMs' textual rationale. Extensive experiments
validate the efficacy of our proposed framework.