PRODIGY: Enabling In-context Learning Over Graphs
Qian Huang Hongyu Ren Peng Chen Gregor Kržmanc Daniel Zeng Percy Liang Jure Leskovec
In-context learning is the ability of a pretrained model to adapt to novel
and diverse downstream tasks by conditioning on prompt examples, without
optimizing any parameters. While large language models have demonstrated this
ability, how in-context learning could be performed over graphs is unexplored.
In this paper, we develop \textbf{Pr}etraining \textbf{O}ver \textbf{D}iverse
\textbf{I}n-Context \textbf{G}raph S\textbf{y}stems (PRODIGY), the first
pretraining framework that enables in-context learning over graphs. The key
idea of our framework is to formulate in-context learning over graphs with a
novel \emph{prompt graph} representation, which connects prompt examples and
queries. We then propose a graph neural network architecture over the prompt
graph and a corresponding family of in-context pretraining objectives. With
PRODIGY, the pretrained model can directly perform novel downstream
classification tasks on unseen graphs via in-context learning. We provide
empirical evidence of the effectiveness of our framework by showcasing its
strong in-context learning performance on tasks involving citation networks and
knowledge graphs. Our approach outperforms the in-context learning accuracy of
contrastive pretraining baselines with hard-coded adaptation by 18\% on average
across all setups. Moreover, it also outperforms standard finetuning with
limited data by 33\% on average with in-context learning.