Masks, Signs, And Learning Rate Rewinding
Advait Gadhikar Rebekka Burkholz
Learning Rate Rewinding (LRR) has been established as a strong variant of
Iterative Magnitude Pruning (IMP) to find lottery tickets in deep
overparameterized neural networks. While both iterative pruning schemes couple
structure and parameter learning, understanding how LRR excels in both aspects
can bring us closer to the design of more flexible deep learning algorithms
that can optimize diverse sets of sparse architectures. To this end, we conduct
experiments that disentangle the effect of mask learning and parameter
optimization and how both benefit from overparameterization. The ability of LRR
to flip parameter signs early and stay robust to sign perturbations seems to
make it not only more effective in mask identification but also in optimizing
diverse sets of masks, including random ones. In support of this hypothesis, we
prove in a simplified single hidden neuron setting that LRR succeeds in more
cases than IMP, as it can escape initially problematic sign configurations.