Reasoning on Efficient Knowledge Paths:Knowledge Graph Guides Large
  Language Model for Domain Question Answering
Yuqi Wang Boran Jiang Yi Luo Dawei He Peng Cheng Liangcai Gao
Large language models (LLMs), such as GPT3.5, GPT4 and LLAMA2 perform
surprisingly well and outperform human experts on many tasks. However, in many
domain-specific evaluations, these LLMs often suffer from hallucination
problems due to insufficient training of relevant corpus. Furthermore,
fine-tuning large models may face problems such as the LLMs are not open source
or the construction of high-quality domain instruction is difficult. Therefore,
structured knowledge databases such as knowledge graph can better provide
domain background knowledge for LLMs and make full use of the reasoning and
analysis capabilities of LLMs. In some previous works, LLM was called multiple
times to determine whether the current triplet was suitable for inclusion in
the subgraph when retrieving subgraphs through a question. Especially for the
question that require a multi-hop reasoning path, frequent calls to LLM will
consume a lot of computing power. Moreover, when choosing the reasoning path,
LLM will be called once for each step, and if one of the steps is selected
incorrectly, it will lead to the accumulation of errors in the following steps.
In this paper, we integrated and optimized a pipeline for selecting reasoning
paths from KG based on LLM, which can reduce the dependency on LLM. In
addition, we propose a simple and effective subgraph retrieval method based on
chain of thought (CoT) and page rank which can returns the paths most likely to
contain the answer. We conduct experiments on three datasets: GenMedGPT-5k
[14], WebQuestions [2], and CMCQA [21]. Finally, RoK can demonstrate that using
fewer LLM calls can achieve the same results as previous SOTAs models.