Synthetic Shifts to Initial Seed Vector Exposes the Brittle Nature of
  Latent-Based Diffusion Models
Mao Po-Yuan Shashank Kotyan Tham Yik Foong Danilo Vasconcellos Vargas
Recent advances in Conditional Diffusion Models have led to substantial
capabilities in various domains. However, understanding the impact of
variations in the initial seed vector remains an underexplored area of concern.
Particularly, latent-based diffusion models display inconsistencies in image
generation under standard conditions when initialized with suboptimal initial
seed vectors. To understand the impact of the initial seed vector on generated
samples, we propose a reliability evaluation framework that evaluates the
generated samples of a diffusion model when the initial seed vector is
subjected to various synthetic shifts. Our results indicate that slight
manipulations to the initial seed vector of the state-of-the-art Stable
Diffusion (Rombach et al., 2022) can lead to significant disturbances in the
generated samples, consequently creating images without the effect of
conditioning variables. In contrast, GLIDE (Nichol et al., 2022) stands out in
generating reliable samples even when the initial seed vector is transformed.
Thus, our study sheds light on the importance of the selection and the impact
of the initial seed vector in the latent-based diffusion model.