Language Model Decoding as Direct Metrics Optimization
Haozhe Ji Pei Ke Hongning Wang Minlie Huang
Despite the remarkable advances in language modeling, current mainstream
decoding methods still struggle to generate texts that align with human texts
across different aspects. In particular, sampling-based methods produce
less-repetitive texts which are often disjunctive in discourse, while
search-based methods maintain topic coherence at the cost of increased
repetition. Overall, these methods fall short in achieving holistic alignment
across a broad range of aspects. In this work, we frame decoding from a
language model as an optimization problem with the goal of strictly matching
the expected performance with human texts measured by multiple metrics of
desired aspects simultaneously. The resulting decoding distribution enjoys an
analytical solution that scales the input language model distribution via a
sequence-level energy function defined by these metrics. And most importantly,
we prove that this induced distribution is guaranteed to improve the perplexity
on human texts, which suggests a better approximation to the underlying
distribution of human texts. To facilitate tractable sampling from this
globally normalized distribution, we adopt the Sampling-Importance-Resampling
technique. Experiments on various domains and model scales demonstrate the
superiority of our method in metrics alignment with human texts and human
evaluation over strong baselines.