GNNBENCH: Fair and Productive Benchmarking for Single-GPU GNN System
Yidong Gong Pradeep Kumar
We hypothesize that the absence of a standardized benchmark has allowed
several fundamental pitfalls in GNN System design and evaluation that the
community has overlooked. In this work, we propose GNNBench, a plug-and-play
benchmarking platform focused on system innovation. GNNBench presents a new
protocol to exchange their captive tensor data, supports custom classes in
System APIs, and allows automatic integration of the same system module to many
deep learning frameworks, such as PyTorch and TensorFlow. To demonstrate the
importance of such a benchmark framework, we integrated several GNN systems.
Our results show that integration with GNNBench helped us identify several
measurement issues that deserve attention from the community.