Explaining generative diffusion models via visual analysis for
  interpretable decision-making process
Ji-Hoon Park Yeong-Joon Ju Seong-Whan Lee
Diffusion models have demonstrated remarkable performance in generation
tasks. Nevertheless, explaining the diffusion process remains challenging due
to it being a sequence of denoising noisy images that are difficult for experts
to interpret. To address this issue, we propose the three research questions to
interpret the diffusion process from the perspective of the visual concepts
generated by the model and the region where the model attends in each time
step. We devise tools for visualizing the diffusion process and answering the
aforementioned research questions to render the diffusion process
human-understandable. We show how the output is progressively generated in the
diffusion process by explaining the level of denoising and highlighting
relationships to foundational visual concepts at each time step through the
results of experiments with various visual analyses using the tools. Throughout
the training of the diffusion model, the model learns diverse visual concepts
corresponding to each time-step, enabling the model to predict varying levels
of visual concepts at different stages. We substantiate our tools using Area
Under Cover (AUC) score, correlation quantification, and cross-attention
mapping. Our findings provide insights into the diffusion process and pave the
way for further research into explainable diffusion mechanisms.