Can Neural Network Memorization Be Localized?
Pratyush Maini Michael C. Mozer Hanie Sedghi Zachary C. Lipton J. Zico Kolter Chiyuan Zhang
Recent efforts at explaining the interplay of memorization and generalization
in deep overparametrized networks have posited that neural networks
$\textit{memorize}$ "hard" examples in the final few layers of the model.
Memorization refers to the ability to correctly predict on $\textit{atypical}$
examples of the training set. In this work, we show that rather than being
confined to individual layers, memorization is a phenomenon confined to a small
set of neurons in various layers of the model. First, via three experimental
sources of converging evidence, we find that most layers are redundant for the
memorization of examples and the layers that contribute to example memorization
are, in general, not the final layers. The three sources are $\textit{gradient
accounting}$ (measuring the contribution to the gradient norms from memorized
and clean examples), $\textit{layer rewinding}$ (replacing specific model
weights of a converged model with previous training checkpoints), and
$\textit{retraining}$ (training rewound layers only on clean examples). Second,
we ask a more generic question: can memorization be localized
$\textit{anywhere}$ in a model? We discover that memorization is often confined
to a small number of neurons or channels (around 5) of the model. Based on
these insights we propose a new form of dropout -- $\textit{example-tied
dropout}$ that enables us to direct the memorization of examples to an apriori
determined set of neurons. By dropping out these neurons, we are able to reduce
the accuracy on memorized examples from $100\%\to3\%$, while also reducing the
generalization gap.