Training Language Models to Generate Text with Citations via
  Fine-grained Rewards
Chengyu Huang Zeqiu Wu Yushi Hu Wenya Wang
While recent Large Language Models (LLMs) have proven useful in answering
user queries, they are prone to hallucination, and their responses often lack
credibility due to missing references to reliable sources. An intuitive
solution to these issues would be to include in-text citations referring to
external documents as evidence. While previous works have directly prompted
LLMs to generate in-text citations, their performances are far from
satisfactory, especially when it comes to smaller LLMs. In this work, we
propose an effective training framework using fine-grained rewards to teach
LLMs to generate highly supportive and relevant citations, while ensuring the
correctness of their responses. We also conduct a systematic analysis of
applying these fine-grained rewards to common LLM training strategies,
demonstrating its advantage over conventional practices. We conduct extensive
experiments on Question Answering (QA) datasets taken from the ALCE benchmark
and validate the model's generalizability using EXPERTQA. On LLaMA-2-7B, the
incorporation of fine-grained rewards achieves the best performance among the
baselines, even surpassing that of GPT-3.5-turbo.