Evaluating Large Language Models on Controlled Generation Tasks
Jiao Sun Yufei Tian Wangchunshu Zhou Nan Xu Qian Hu Rahul Gupta John Frederick Wieting Nanyun Peng Xuezhe Ma
While recent studies have looked into the abilities of large language models
in various benchmark tasks, including question generation, reading
comprehension, multilingual and etc, there have been few studies looking into
the controllability of large language models on generation tasks. We present an
extensive analysis of various benchmarks including a sentence planning
benchmark with different granularities. After comparing large language models
against state-of-the-start finetuned smaller models, we present a spectrum
showing large language models falling behind, are comparable, or exceed the
ability of smaller models. We conclude that **large language models struggle at
meeting fine-grained hard constraints**.