VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor
  Cores
Roberto L. Castro Andrei Ivanov Diego Andrade Tal Ben-Nun Basilio B. Fraguela Torsten Hoefler
The increasing success and scaling of Deep Learning models demands higher
computational efficiency and power. Sparsification can lead to both smaller
models as well as higher compute efficiency, and accelerated hardware is
becoming available. However, exploiting it efficiently requires kernel
implementations, pruning algorithms, and storage formats, to utilize hardware
support of specialized sparse vector units. An example of those are the
NVIDIA's Sparse Tensor Cores (SPTCs), which promise a 2x speedup. However,
SPTCs only support the 2:4 format, limiting achievable sparsity ratios to 50%.
We present the V:N:M format, which enables the execution of arbitrary N:M
ratios on SPTCs. To efficiently exploit the resulting format, we propose
Spatha, a high-performance sparse-library for DL routines. We show that Spatha
achieves up to 37x speedup over cuBLAS. We also demonstrate a second-order
pruning technique that enables sparsification to high sparsity ratios with
V:N:M and little to no loss in accuracy in modern transformers.