Recurrent Neural Networks (RNNs) are powerful tools for solving sequence-based problems, but their efficacy and execution time are dependent on the size of the network.   Following recent work in simplifying these networks with model pruning and a novel mapping of work onto GPUs, we design an efficient implementation for sparse RNNs.   We investigate several optimizations and tradeoffs: Lamport timestamps, wide memory loads, and a bank-aware weight layout.   With these optimizations, we achieve speedups of over 6x over the next best algorithm for a hidden layer of size 2304, batch size of 4, and a density of 30%.   Further, our technique allows for models of over 5x the size to fit on a GPU for a speedup of 2x, enabling larger networks to help advance the state-of-the-art.   We perform case studies on NMT and speech recognition tasks in the appendix, accelerating their recurrent layers by up to 3x. Many sequence-based problems, including Speech Recognition BID0 and Neural Machine Translation (NMT) BID3 , can be solved effectively with Recurrent Neural Networks (RNNs). BID2 showed that these networks can run efficiently on massively parallel processors such as GPUs, and BID8 found that if the network is small enough to fit in the register file of a GPU, a persistent approach can be used to increase performance.In parallel, many network compression methods BID16 BID12 have been shown to reduce the model size of both Convolutional Neural Networks (CNNs) and RNNs. Recent work in this area has found that model pruning, in particular, can lead to significant reductions in the number of important network parameters for RNNs BID31 We present an approach that combines both of these techniques into an efficient and expandable approach. In particular, our work makes the following contributions:• Larger sparse RNNs can be run more efficiently on GPUs.• Sparse RNNs can be run with smaller batch sizes more efficiently on GPUs.• Various optimizations on top of the naïve implementation, necessary to achieve high performance.• Case studies using our technique showing 1 ) generalization to LSTMs, 2) practical network design considerations, and 3) speedups of up to 3× on two non-synthetic workloads.A naïve implementation of the idea, presented in Section 3, leads to limited benefit; we present a series of optimizations in Section 4 that help to achieve a high level of performance. Section 5 describes the experimental setup and results, and we discuss future work and our conclusions in Sections 6 and 7. The appendix presents a case study on a machine translation task.2 RELATED WORK 2.1 RNNS Recurrent Neural Networks (RNNs) are powerful tools for solving series-based problems, such as NMT BID3 BID18 BID0 BID26 , language modeling BID27 , and various NLP tasks BID7 . More complex recurrent networks have been devised to build on the basic RNN structure, such as Long/Short Term Memory networks (LSTMs) BID20 and Gated Recurrent Units (GRUs) BID6 . Though we focus on RNNs in this work for simplicity, our approach can extend to other recurrent network types, such as the LSTMs used in our case study. In particular, we build on a recent GPU-based method storing recurrent weights on-chip BID8 . Sparse persistent RNNs increase the performance of pruned networks, but without a series of optimizations we described in Section 4, the improvement is much lower. An optimized implementation is critical to achieve peak performance. After these optimizations, we see that our approach works best for sparser workloads, and, for most layer sizes, 10% density is sufficient to beat all other approaches, though for smaller layer sizes (up to 2304), our approach is the winner even up to a density of 30%. Different batch sizes can affect the efficiency of all techniques; in general, dense GEMMs and our approach scale the best, which tempers the benefit offered by other algorithms. Finally, the number of timesteps is largely immaterial to the relative performance of the various techniques.We have described an efficient algorithm for recurrent layers. To exploit this algorithm, one must first prune the recurrent layers of a target network. We perform this pruning during the training process for a nureal machine translation network BID22 , and we look to past work ) that has pruned a commercial speech recognition network, Deep Speech 2 BID0 . The results on these real workloads are promising: for a given accuracy, we show a speedup of between 2.0× to 3.1× in the machine translation network's recurrent layers pruned with load-balancing in mind, and a speedup of 1.2× to 2.9 on Deep Speech 2's recurrent layers pruned without load-balancing. Please refer to Appendix B for more details about the training processes and a full discussion of the results. We introduced sparse persistent RNNs, an efficient new algorithm for accelerating pruned recurrent networks. Further, we explored several optimizations that were needed to achieve these results on V100, a recent GPU. Our optimized technique allows for 7.3×, 3.4×, and 1.8× speedups against dense GEMM, sparse GEMM, and dense persistent implementations for a hidden layer of size 1792, batch size of 4, and a density of 10%. We show that much larger networks can be deployed onto a GPU of a fixed size with performance increases of around 5× over the next best solution for a density of 1-30% on a layer size of 2304; notably, this sparsity range includes denser workloads than typically perform worse with sparse optimizations. We also show promising results on much larger layers -7168 and 11520 achieve speedups of 1.9× for 2.6% and 1% densities, respectively. Our approach speeds up pruned NMT and speech recognition networks' recurrent layers by up to 3×. Finally, load-balanced pruning can significantly improve a network's throughput, and our technique is necessary to achieve both high performance and accuracy in some recurrent layers, as detailed in the case studies in our appendix. Without consideration of the accuracy of pruned networks and their execution speed (on state-of-theart algorithms) together, the conclusion that a large, sparse network is better than a small, dense one is not sufficiently proven. Our technique and these case studies show, among these insights, that this can be the case, however:• Our technique extends to LSTMs with little effort.• Our technique allows for pruned recurrent layers to run more efficiently than with any other existing algorithm.• Naïve pruning is not necessarily better than pruning with load-balancing in mind, especially when considering achievable performance.• Like past work, we see that a larger, sparse network can be more accurate than a smaller, dense one.• Comparisons against persistent kernels, which can beat non-resident sparse approaches,show that for a given accuracy or performance target, pruning a network and using our technique is the best choice.
Combining network pruning and persistent kernels into a practical, fast, and accurate network implementation. This paper introduces sparse persistent RNNs, a mechanism to add pruning to the existing work of stashing RNN weights on a chip.