Quantifying lottery tickets under label noise: accuracy, calibration,
  and complexity
Viplove Arora Daniele Irto Sebastian Goldt Guido Sanguinetti
Pruning deep neural networks is a widely used strategy to alleviate the
computational burden in machine learning. Overwhelming empirical evidence
suggests that pruned models retain very high accuracy even with a tiny fraction
of parameters. However, relatively little work has gone into characterising the
small pruned networks obtained, beyond a measure of their accuracy. In this
paper, we use the sparse double descent approach to identify univocally and
characterise pruned models associated with classification tasks. We observe
empirically that, for a given task, iterative magnitude pruning (IMP) tends to
converge to networks of comparable sizes even when starting from full networks
with sizes ranging over orders of magnitude. We analyse the best pruned models
in a controlled experimental setup and show that their number of parameters
reflects task difficulty and that they are much better than full networks at
capturing the true conditional probability distribution of the labels. On real
data, we similarly observe that pruned models are less prone to overconfident
predictions. Our results suggest that pruned models obtained via IMP not only
have advantageous computational properties but also provide a better
representation of uncertainty in learning.