How faithful are RAG models? Quantifying the tug-of-war between RAG and
  LLMs' internal prior
Kevin Wu Eric Wu James Zou
Retrieval augmented generation (RAG) is often used to fix hallucinations and
provide up-to-date knowledge for large language models (LLMs). However, in
cases when the LLM alone incorrectly answers a question, does providing the
correct retrieved content always fix the error? Conversely, in cases where the
retrieved content is incorrect, does the LLM know to ignore the wrong
information, or does it recapitulate the error? To answer these questions, we
systematically analyze the tug-of-war between a LLM's internal knowledge (i.e.
its prior) and the retrieved information in settings when they disagree. We
test GPT-4 and other LLMs on question-answering abilities across datasets with
and without reference documents. As expected, providing the correct retrieved
information fixes most model mistakes (94% accuracy). However, when the
reference document is perturbed with increasing levels of wrong values, the LLM
is more likely to recite the incorrect, modified information when its internal
prior is weaker but is more resistant when its prior is stronger. Similarly, we
also find that the more the modified information deviates from the model's
prior, the less likely the model is to prefer it. These results highlight an
underlying tension between a model's prior knowledge and the information
presented in reference documents.