Is Model Collapse Inevitable? Breaking the Curse of Recursion by
  Accumulating Real and Synthetic Data
Matthias Gerstgrasser Rylan Schaeffer Apratim Dey Rafael Rafailov Henry Sleight John Hughes Tomasz Korbak Rajashree Agrawal Dhruv Pai Andrey Gromov Daniel A. Roberts Diyi Yang David L. Donoho Sanmi Koyejo
The proliferation of generative models, combined with pretraining on
web-scale data, raises a timely question: what happens when these models are
trained on their own generated outputs? Recent investigations into model-data
feedback loops discovered that such loops can lead to model collapse, a
phenomenon where performance progressively degrades with each model-fitting
iteration until the latest model becomes useless. However, several recent
papers studying model collapse assumed that new data replace old data over time
rather than assuming data accumulate over time. In this paper, we compare these
two settings and show that accumulating data prevents model collapse. We begin
by studying an analytically tractable setup in which a sequence of linear
models are fit to the previous models' predictions. Previous work showed if
data are replaced, the test error increases linearly with the number of
model-fitting iterations; we extend this result by proving that if data instead
accumulate, the test error has a finite upper bound independent of the number
of iterations. We next empirically test whether accumulating data similarly
prevents model collapse by pretraining sequences of language models on text
corpora. We confirm that replacing data does indeed cause model collapse, then
demonstrate that accumulating data prevents model collapse; these results hold
across a range of model sizes, architectures and hyperparameters. We further
show that similar results hold for other deep generative models on real data:
diffusion models for molecule generation and variational autoencoders for image
generation. Our work provides consistent theoretical and empirical evidence
that data accumulation mitigates model collapse.