Learning to Compose SuperWeights for Neural Parameter Allocation Search
Piotr Teterwak Soren Nelson Nikoli Dryden Dina Bashkirova Kate Saenko Bryan A. Plummer
Neural parameter allocation search (NPAS) automates parameter sharing by
obtaining weights for a network given an arbitrary, fixed parameter budget.
Prior work has two major drawbacks we aim to address. First, there is a
disconnect in the sharing pattern between the search and training steps, where
weights are warped for layers of different sizes during the search to measure
similarity, but not during training, resulting in reduced performance. To
address this, we generate layer weights by learning to compose sets of
SuperWeights, which represent a group of trainable parameters. These
SuperWeights are created to be large enough so they can be used to represent
any layer in the network, but small enough that they are computationally
efficient. The second drawback we address is the method of measuring similarity
between shared parameters. Whereas prior work compared the weights themselves,
we argue this does not take into account the amount of conflict between the
shared weights. Instead, we use gradient information to identify layers with
shared weights that wish to diverge from each other. We demonstrate that our
SuperWeight Networks consistently boost performance over the state-of-the-art
on the ImageNet and CIFAR datasets in the NPAS setting. We further show that
our approach can generate parameters for many network architectures using the
same set of weights. This enables us to support tasks like efficient ensembling
and anytime prediction, outperforming fully-parameterized ensembles with 17%
fewer parameters.