Knowledge-Augmented Language Model Verification
Jinheon Baek Soyeong Jeong Minki Kang Jong C. Park Sung Ju Hwang
Recent Language Models (LMs) have shown impressive capabilities in generating
texts with the knowledge internalized in parameters. Yet, LMs often generate
the factually incorrect responses to the given queries, since their knowledge
may be inaccurate, incomplete, and outdated. To address this problem, previous
works propose to augment LMs with the knowledge retrieved from an external
knowledge source. However, such approaches often show suboptimal text
generation performance due to two reasons: 1) the model may fail to retrieve
the knowledge relevant to the given query, or 2) the model may not faithfully
reflect the retrieved knowledge in the generated text. To overcome these, we
propose to verify the output and the knowledge of the knowledge-augmented LMs
with a separate verifier, which is a small LM that is trained to detect those
two types of errors through instruction-finetuning. Then, when the verifier
recognizes an error, we can rectify it by either retrieving new knowledge or
generating new text. Further, we use an ensemble of the outputs from different
instructions with a single verifier to enhance the reliability of the
verification processes. We validate the effectiveness of the proposed
verification steps on multiple question answering benchmarks, whose results
show that the proposed verifier effectively identifies retrieval and generation
errors, allowing LMs to provide more factually correct outputs. Our code is
available at https://github.com/JinheonBaek/KALMV.