InfoDiffusion: Information Entropy Aware Diffusion Process for
  Non-Autoregressive Text Generation
Renzhi Wang Jing Li Piji Li
Diffusion models have garnered considerable interest in the field of text
generation. Several studies have explored text diffusion models with different
structures and applied them to various tasks, including named entity
recognition and summarization. However, there exists a notable disparity
between the "easy-first" text generation process of current diffusion models
and the "keyword-first" natural text generation process of humans, which has
received limited attention. To bridge this gap, we propose InfoDiffusion, a
non-autoregressive text diffusion model. Our approach introduces a
"keyinfo-first" generation strategy and incorporates a noise schedule based on
the amount of text information. In addition, InfoDiffusion combines
self-conditioning with a newly proposed partially noising model structure.
Experimental results show that InfoDiffusion outperforms the baseline model in
terms of generation quality and diversity, as well as exhibiting higher
sampling efficiency.