A Negative Result on Gradient Matching for Selective Backprop
Lukas Balles Cedric Archambeau Giovanni Zappella
With increasing scale in model and dataset size, the training of deep neural
networks becomes a massive computational burden. One approach to speed up the
training process is Selective Backprop. For this approach, we perform a forward
pass to obtain a loss value for each data point in a minibatch. The backward
pass is then restricted to a subset of that minibatch, prioritizing high-loss
examples. We build on this approach, but seek to improve the subset selection
mechanism by choosing the (weighted) subset which best matches the mean
gradient over the entire minibatch. We use the gradients w.r.t. the model's
last layer as a cheap proxy, resulting in virtually no overhead in addition to
the forward pass. At the same time, for our experiments we add a simple random
selection baseline which has been absent from prior work. Surprisingly, we find
that both the loss-based as well as the gradient-matching strategy fail to
consistently outperform the random baseline.