iSpLib: A Library for Accelerating Graph Neural Networks using
  Auto-tuned Sparse Operations
Md Saidul Hoque Anik Pranav Badhe Rohit Gampa Ariful Azad
Core computations in Graph Neural Network (GNN) training and inference are
often mapped to sparse matrix operations such as sparse-dense matrix
multiplication (SpMM). These sparse operations are harder to optimize by manual
tuning because their performance depends significantly on the sparsity of input
graphs, GNN models, and computing platforms. To address this challenge, we
present iSpLib, a PyTorch-based C++ library equipped with auto-tuned sparse
operations. iSpLib expedites GNN training with a cache-enabled backpropagation
that stores intermediate matrices in local caches. The library offers a
user-friendly Python plug-in that allows users to take advantage of our
optimized PyTorch operations out-of-the-box for any existing linear
algebra-based PyTorch implementation of popular GNNs (Graph Convolution
Network, GraphSAGE, Graph Inference Network, etc.) with only two lines of
additional code. We demonstrate that iSpLib obtains up to 27x overall training
speedup compared to the equivalent PyTorch 2.1.0 and PyTorch Geometric 2.4.0
implementations on the CPU. Our library is publicly available at
https://github.com/HipGraph/iSpLib (https://doi.org/10.5281/zenodo.10806511).