Can GNN be Good Adapter for LLMs?
Xuanwen Huang Kaiqiao Han Yang Yang Dezheng Bao Quanjin Tao Ziwei Chai Qi Zhu
Recently, large language models (LLMs) have demonstrated superior
capabilities in understanding and zero-shot learning on textual data, promising
significant advances for many text-related domains. In the graph domain,
various real-world scenarios also involve textual data, where tasks and node
features can be described by text. These text-attributed graphs (TAGs) have
broad applications in social media, recommendation systems, etc. Thus, this
paper explores how to utilize LLMs to model TAGs. Previous methods for TAG
modeling are based on million-scale LMs. When scaled up to billion-scale LLMs,
they face huge challenges in computational costs. Additionally, they also
ignore the zero-shot inference capabilities of LLMs. Therefore, we propose
GraphAdapter, which uses a graph neural network (GNN) as an efficient adapter
in collaboration with LLMs to tackle TAGs. In terms of efficiency, the GNN
adapter introduces only a few trainable parameters and can be trained with low
computation costs. The entire framework is trained using auto-regression on
node text (next token prediction). Once trained, GraphAdapter can be seamlessly
fine-tuned with task-specific prompts for various downstream tasks. Through
extensive experiments across multiple real-world TAGs, GraphAdapter based on
Llama 2 gains an average improvement of approximately 5\% in terms of node
classification. Furthermore, GraphAdapter can also adapt to other language
models, including RoBERTa, GPT-2. The promising results demonstrate that GNNs
can serve as effective adapters for LLMs in TAG modeling.