On the Scalability of GNNs for Molecular Graphs
Maciej Sypetkowski Frederik Wenkel Farimah Poursafaei Nia Dickson Karush Suri Philip Fradkin Dominique Beaini
Scaling deep learning models has been at the heart of recent revolutions in
language modelling and image generation. Practitioners have observed a strong
relationship between model size, dataset size, and performance. However,
structure-based architectures such as Graph Neural Networks (GNNs) are yet to
show the benefits of scale mainly due to the lower efficiency of sparse
operations, large data requirements, and lack of clarity about the
effectiveness of various architectures. We address this drawback of GNNs by
studying their scaling behavior. Specifically, we analyze message-passing
networks, graph Transformers, and hybrid architectures on the largest public
collection of 2D molecular graphs. For the first time, we observe that GNNs
benefit tremendously from the increasing scale of depth, width, number of
molecules, number of labels, and the diversity in the pretraining datasets,
resulting in a 30.25% improvement when scaling to 1 billion parameters and
28.98% improvement when increasing size of dataset to eightfold. We further
demonstrate strong finetuning scaling behavior on 38 tasks, outclassing
previous large models. We hope that our work paves the way for an era where
foundational GNNs drive pharmaceutical drug discovery.