Dense Hopfield Networks in the Teacher-Student Setting
Robin Thériault Daniele Tantari
Dense Hopfield networks are known for their feature to prototype transition
and adversarial robustness. However, previous theoretical studies have been
mostly concerned with their storage capacity. We bridge this gap by studying
the phase diagram of p-body Hopfield networks in the teacher-student setting of
an unsupervised learning problem, uncovering ferromagnetic phases reminiscent
of the prototype and feature learning regimes. On the Nishimori line, we find
the critical size of the training set necessary for efficient pattern
retrieval. Interestingly, we find that that the paramagnetic to ferromagnetic
transition of the teacher-student setting coincides with the paramagnetic to
spin-glass transition of the direct model, i.e. with random patterns. Outside
of the Nishimori line, we investigate the learning performance in relation to
the inference temperature and dataset noise. Moreover, we show that using a
larger p for the student than the teacher gives the student an extensive
tolerance to noise. We then derive a closed-form expression measuring the
adversarial robustness of such a student at zero temperature, corroborating the
positive correlation between number of parameters and robustness observed in
large neural networks. We also use our model to clarify why the prototype phase
of modern Hopfield networks is adversarially robust.