Memory in Plain Sight: A Survey of the Uncanny Resemblances between
  Diffusion Models and Associative Memories
Benjamin Hoover Hendrik Strobelt Dmitry Krotov Judy Hoffman Zsolt Kira Duen Horng Chau
Diffusion Models (DMs) have recently set state-of-the-art on many generation
benchmarks. However, there are myriad ways to describe them mathematically,
which makes it difficult to develop a simple understanding of how they work. In
this survey, we provide a concise overview of DMs from the perspective of
dynamical systems and Ordinary Differential Equations (ODEs) which exposes a
mathematical connection to the highly related yet often overlooked class of
energy-based models, called Associative Memories (AMs). Energy-based AMs are a
theoretical framework that behave much like denoising DMs, but they enable us
to directly compute a Lyapunov energy function on which we can perform gradient
descent to denoise data. We then summarize the 40 year history of energy-based
AMs, beginning with the original Hopfield Network, and discuss new research
directions for AMs and DMs that are revealed by characterizing the extent of
their similarities and differences