Neural Scaling Laws on Graphs
Jingzhe Liu Haitao Mao Zhikai Chen Tong Zhao Neil Shah Jiliang Tang
Deep graph models (e.g., graph neural networks and graph transformers) have
become important techniques for leveraging knowledge across various types of
graphs. Yet, the scaling properties of deep graph models have not been
systematically investigated, casting doubt on the feasibility of achieving
large graph models through enlarging the model and dataset sizes. In this work,
we delve into neural scaling laws on graphs from both model and data
perspectives. We first verify the validity of such laws on graphs, establishing
formulations to describe the scaling behaviors. For model scaling, we
investigate the phenomenon of scaling law collapse and identify overfitting as
the potential reason. Moreover, we reveal that the model depth of deep graph
models can impact the model scaling behaviors, which differ from observations
in other domains such as CV and NLP. For data scaling, we suggest that the
number of graphs can not effectively metric the graph data volume in scaling
law since the sizes of different graphs are highly irregular. Instead, we
reform the data scaling law with the number of edges as the metric to address
the irregular graph sizes. We further demonstrate the reformed law offers a
unified view of the data scaling behaviors for various fundamental graph tasks
including node classification, link prediction, and graph classification. This
work provides valuable insights into neural scaling laws on graphs, which can
serve as an essential step toward large graph models.