A comprehensive study of spike and slab shrinkage priors for
  structurally sparse Bayesian neural networks
Sanket Jantre Shrijita Bhattacharya Tapabrata Maiti
Network complexity and computational efficiency have become increasingly
significant aspects of deep learning. Sparse deep learning addresses these
challenges by recovering a sparse representation of the underlying target
function by reducing heavily over-parameterized deep neural networks.
Specifically, deep neural architectures compressed via structured sparsity
(e.g. node sparsity) provide low latency inference, higher data throughput, and
reduced energy consumption. In this paper, we explore two well-established
shrinkage techniques, Lasso and Horseshoe, for model compression in Bayesian
neural networks. To this end, we propose structurally sparse Bayesian neural
networks which systematically prune excessive nodes with (i) Spike-and-Slab
Group Lasso (SS-GL), and (ii) Spike-and-Slab Group Horseshoe (SS-GHS) priors,
and develop computationally tractable variational inference including
continuous relaxation of Bernoulli variables. We establish the contraction
rates of the variational posterior of our proposed models as a function of the
network topology, layer-wise node cardinalities, and bounds on the network
weights. We empirically demonstrate the competitive performance of our models
compared to the baseline models in prediction accuracy, model compression, and
inference latency.