A Knowledge-Injected Curriculum Pretraining Framework for Question
  Answering
Xin Lin Tianhuang Su Zhenya Huang Shangzi Xue Haifeng Liu Enhong Chen
Knowledge-based question answering (KBQA) is a key task in NLP research, and
also an approach to access the web data and knowledge, which requires
exploiting knowledge graphs (KGs) for reasoning. In the literature, one
promising solution for KBQA is to incorporate the pretrained language model
(LM) with KGs by generating KG-centered pretraining corpus, which has shown its
superiority. However, these methods often depend on specific techniques and
resources to work, which may not always be available and restrict its
application. Moreover, existing methods focus more on improving language
understanding with KGs, while neglect the more important human-like complex
reasoning. To this end, in this paper, we propose a general Knowledge-Injected
Curriculum Pretraining framework (KICP) to achieve comprehensive KG learning
and exploitation for KBQA tasks, which is composed of knowledge injection (KI),
knowledge adaptation (KA) and curriculum reasoning (CR). Specifically, the KI
module first injects knowledge into the LM by generating KG-centered
pretraining corpus, and generalizes the process into three key steps that could
work with different implementations for flexible application. Next, the KA
module learns knowledge from the generated corpus with LM equipped with an
adapter as well as keeps its original natural language understanding ability to
reduce the negative impacts of the difference between the generated and natural
corpus. Last, to enable the LM with complex reasoning, the CR module follows
human reasoning patterns to construct three corpora with increasing
difficulties of reasoning, and further trains the LM from easy to hard in a
curriculum manner. We provide an implementation of the general framework, and
evaluate the proposed KICP on four real-word datasets. The results demonstrate
that our framework can achieve higher performances.