An End-to-End Network Pruning Pipeline with Sparsity Enforcement
Evan Dogariu
Neural networks have emerged as a powerful tool for solving complex tasks
across various domains, but their increasing size and computational
requirements have posed significant challenges in deploying them on
resource-constrained devices. Neural network sparsification, and in particular
pruning, has emerged as an effective technique to alleviate these challenges by
reducing model size, computational complexity, and memory footprint while
maintaining competitive performance. However, many pruning pipelines modify the
standard training pipeline at only a single stage, if at all. In this work, we
look to develop an end-to-end training pipeline that befits neural network
pruning and sparsification at all stages of training. To do so, we make use of
nonstandard model parameter initialization, pre-pruning training methodologies,
and post-pruning training optimizations. We conduct experiments utilizing
combinations of these methods, in addition to different techniques used in the
pruning step, and find that our combined pipeline can achieve significant gains
over current state of the art approaches to neural network sparsification.