KITLM: Domain-Specific Knowledge InTegration into Language Models for
  Question Answering
Ankush Agarwal Sakharam Gawade Amar Prakash Azad Pushpak Bhattacharyya
Large language models (LLMs) have demonstrated remarkable performance in a
wide range of natural language tasks. However, as these models continue to grow
in size, they face significant challenges in terms of computational costs.
Additionally, LLMs often lack efficient domain-specific understanding, which is
particularly crucial in specialized fields such as aviation and healthcare. To
boost the domain-specific understanding, we propose, KITLM, a novel knowledge
base integration approach into language model through relevant information
infusion. By integrating pertinent knowledge, not only the performance of the
language model is greatly enhanced, but the model size requirement is also
significantly reduced while achieving comparable performance. Our proposed
knowledge-infused model surpasses the performance of both GPT-3.5-turbo and the
state-of-the-art knowledge infusion method, SKILL, achieving over 1.5 times
improvement in exact match scores on the MetaQA. KITLM showed a similar
performance boost in the aviation domain with AeroQA. The drastic performance
improvement of KITLM over the existing methods can be attributed to the
infusion of relevant knowledge while mitigating noise. In addition, we release
two curated datasets to accelerate knowledge infusion research in specialized
fields: a) AeroQA, a new benchmark dataset designed for multi-hop
question-answering within the aviation domain, and b) Aviation Corpus, a
dataset constructed from unstructured text extracted from the National
Transportation Safety Board reports. Our research contributes to advancing the
field of domain-specific language understanding and showcases the potential of
knowledge infusion techniques in improving the performance of language models
on question-answering.