Input margins can predict generalization too
Coenraad Mouton Marthinus W. Theunissen Marelie H. Davel
Understanding generalization in deep neural networks is an active area of
research. A promising avenue of exploration has been that of margin
measurements: the shortest distance to the decision boundary for a given sample
or its representation internal to the network. While margins have been shown to
be correlated with the generalization ability of a model when measured at its
hidden representations (hidden margins), no such link between large margins and
generalization has been established for input margins. We show that while input
margins are not generally predictive of generalization, they can be if the
search space is appropriately constrained. We develop such a measure based on
input margins, which we refer to as `constrained margins'. The predictive power
of this new measure is demonstrated on the 'Predicting Generalization in Deep
Learning' (PGDL) dataset and contrasted with hidden representation margins. We
find that constrained margins achieve highly competitive scores and outperform
other margin measurements in general. This provides a novel insight on the
relationship between generalization and classification margins, and highlights
the importance of considering the data manifold for investigations of
generalization in DNNs.