Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A
  Case Study on Domain-Specific Queries in Private Knowledge-Bases
Jiarui Li Ye Yuan Zehua Zhang
We proposed an end-to-end system design towards utilizing Retrieval Augmented
Generation (RAG) to improve the factual accuracy of Large Language Models
(LLMs) for domain-specific and time-sensitive queries related to private
knowledge-bases. Our system integrates RAG pipeline with upstream datasets
processing and downstream performance evaluation. Addressing the challenge of
LLM hallucinations, we finetune models with a curated dataset which originates
from CMU's extensive resources and annotated with the teacher model. Our
experiments demonstrate the system's effectiveness in generating more accurate
answers to domain-specific and time-sensitive inquiries. The results also
revealed the limitations of fine-tuning LLMs with small-scale and skewed
datasets. This research highlights the potential of RAG systems in augmenting
LLMs with external datasets for improved performance in knowledge-intensive
tasks. Our code and models are available on Github.