Likelihood-Based Diffusion Language Models
Ishaan Gulrajani Tatsunori B. Hashimoto
Despite a growing interest in diffusion-based language models, existing work
has not shown that these models can attain nontrivial likelihoods on standard
language modeling benchmarks. In this work, we take the first steps towards
closing the likelihood gap between autoregressive and diffusion-based language
models, with the goal of building and releasing a diffusion model which
outperforms a small but widely-known autoregressive model. We pursue this goal
through algorithmic improvements, scaling laws, and increased compute. On the
algorithmic front, we introduce several methodological improvements for the
maximum-likelihood training of diffusion language models. We then study scaling
laws for our diffusion models and find compute-optimal training regimes which
differ substantially from autoregressive models. Using our methods and scaling
analysis, we train and release Plaid 1B, a large diffusion language model which
outperforms GPT-2 124M in likelihood on benchmark datasets and generates fluent
samples in unconditional and zero-shot control settings.