Measuring Sharpness in Grokking
Jack Miller Patrick Gleeson Charles O'Neill Thang Bui Noam Levi
Neural networks sometimes exhibit grokking, a phenomenon where perfect or
near-perfect performance is achieved on a validation set well after the same
performance has been obtained on the corresponding training set. In this
workshop paper, we introduce a robust technique for measuring grokking, based
on fitting an appropriate functional form. We then use this to investigate the
sharpness of transitions in training and validation accuracy under two
settings. The first setting is the theoretical framework developed by Levi et
al. (2023) where closed form expressions are readily accessible. The second
setting is a two-layer MLP trained to predict the parity of bits, with grokking
induced by the concealment strategy of Miller et al. (2023). We find that
trends between relative grokking gap and grokking sharpness are similar in both
settings when using absolute and relative measures of sharpness. Reflecting on
this, we make progress toward explaining some trends and identify the need for
further study to untangle the various mechanisms which influence the sharpness
of grokking.