Frame by Familiar Frame: Understanding Replication in Video Diffusion
  Models
Aimon Rahman Malsha V. Perera Vishal M. Patel
Building on the momentum of image generation diffusion models, there is an
increasing interest in video-based diffusion models. However, video generation
poses greater challenges due to its higher-dimensional nature, the scarcity of
training data, and the complex spatiotemporal relationships involved. Image
generation models, due to their extensive data requirements, have already
strained computational resources to their limits. There have been instances of
these models reproducing elements from the training samples, leading to
concerns and even legal disputes over sample replication. Video diffusion
models, which operate with even more constrained datasets and are tasked with
generating both spatial and temporal content, may be more prone to replicating
samples from their training sets. Compounding the issue, these models are often
evaluated using metrics that inadvertently reward replication. In our paper, we
present a systematic investigation into the phenomenon of sample replication in
video diffusion models. We scrutinize various recent diffusion models for video
synthesis, assessing their tendency to replicate spatial and temporal content
in both unconditional and conditional generation scenarios. Our study
identifies strategies that are less likely to lead to replication. Furthermore,
we propose new evaluation strategies that take replication into account,
offering a more accurate measure of a model's ability to generate the original
content.