Semantic Approach to Quantifying the Consistency of Diffusion Model
  Image Generation
Brinnae Bent
In this study, we identify the need for an interpretable, quantitative score
of the repeatability, or consistency, of image generation in diffusion models.
We propose a semantic approach, using a pairwise mean CLIP (Contrastive
Language-Image Pretraining) score as our semantic consistency score. We applied
this metric to compare two state-of-the-art open-source image generation
diffusion models, Stable Diffusion XL and PixArt-{\alpha}, and we found
statistically significant differences between the semantic consistency scores
for the models. Agreement between the Semantic Consistency Score selected model
and aggregated human annotations was 94%. We also explored the consistency of
SDXL and a LoRA-fine-tuned version of SDXL and found that the fine-tuned model
had significantly higher semantic consistency in generated images. The Semantic
Consistency Score proposed here offers a measure of image generation alignment,
facilitating the evaluation of model architectures for specific tasks and
aiding in informed decision-making regarding model selection.