Grokking in Linear Estimators -- A Solvable Model that Groks without
  Understanding
Noam Levi Alon Beck Yohai Bar-Sinai
Grokking is the intriguing phenomenon where a model learns to generalize long
after it has fit the training data. We show both analytically and numerically
that grokking can surprisingly occur in linear networks performing linear tasks
in a simple teacher-student setup with Gaussian inputs. In this setting, the
full training dynamics is derived in terms of the training and generalization
data covariance matrix. We present exact predictions on how the grokking time
depends on input and output dimensionality, train sample size, regularization,
and network initialization. We demonstrate that the sharp increase in
generalization accuracy may not imply a transition from "memorization" to
"understanding", but can simply be an artifact of the accuracy measure. We
provide empirical verification for our calculations, along with preliminary
results indicating that some predictions also hold for deeper networks, with
non-linear activations.