Towards a Unified Language Model for Knowledge-Intensive Tasks Utilizing
  External Corpus
Xiaoxi Li Zhicheng Dou Yujia Zhou Fangchao Liu
The advent of large language models (LLMs) has showcased their efficacy
across various domains, yet they often hallucinate, especially in
knowledge-intensive tasks that require external knowledge sources. To improve
factual accuracy of language models, retrieval-augmented generation (RAG) has
emerged as a popular solution. However, traditional retrieval modules often
rely on large-scale document indexes, which can be disconnected from generative
tasks. Through generative retrieval (GR) approach, language models can achieve
superior retrieval performance by directly generating relevant document
identifiers (DocIDs). However, the relationship between GR and downstream
tasks, as well as the potential of LLMs in GR, remains unexplored. In this
paper, we present a unified language model that utilizes external corpus to
handle various knowledge-intensive tasks by seamlessly integrating generative
retrieval, closed-book generation, and RAG. In order to achieve effective
retrieval and generation through a unified continuous decoding process, we
introduce the following mechanisms: (1) a ranking-oriented DocID decoding
strategy, which improves ranking ability by directly learning from a DocID
ranking list; (2) a continuous generation strategy to facilitate effective and
efficient RAG; (3) well-designed auxiliary DocID understanding tasks to enhance
the model's comprehension of DocIDs and their relevance to downstream tasks.
Our approach is evaluated on the widely used KILT benchmark using two variants
of backbone models: an encoder-decoder T5 model and a decoder-only LLM, Llama2.
Experimental results showcase the superior performance of our models in both
retrieval and downstream knowledge-intensive tasks.