Accelerating Transformer Pre-Training with 2:4 Sparsity
Yuezhou Hu Kang Zhao Weiyu Huang Jianfei Chen Jun Zhu
Training large Transformers is slow, but recent innovations on GPU
architecture gives us an advantage. NVIDIA Ampere GPUs can execute a
fine-grained 2:4 sparse matrix multiplication twice as fast as its dense
equivalent. In the light of this property, we comprehensively investigate the
feasibility of accelerating feed-forward networks (FFNs) of Transformers in
pre-training. First, we define a "flip rate" to monitor the stability of a 2:4
training process. Utilizing this metric, we suggest two techniques to preserve
accuracy: to modify the sparse-refined straight-through estimator by applying
the mask decay term on gradients, and to enhance the model's quality by a
simple yet effective dense fine-tuning procedure near the end of pre-training.
Besides, we devise two effective techniques to practically accelerate training:
to calculate transposable 2:4 mask by convolution, and to accelerate gated
activation functions by reducing GPU L2 cache miss. Experiments show that a
combination of our methods reaches the best performance on multiple
Transformers among different 2:4 training methods, while actual acceleration
can be observed on different shapes of Transformer block.