Bridging the KB-Text Gap: Leveraging Structured Knowledge-aware
  Pre-training for KBQA
Guanting Dong Rumei Li Sirui Wang Yupeng Zhang Yunsen Xian Weiran Xu
Knowledge Base Question Answering (KBQA) aims to answer natural language
questions with factual information such as entities and relations in KBs.
However, traditional Pre-trained Language Models (PLMs) are directly
pre-trained on large-scale natural language corpus, which poses challenges for
them in understanding and representing complex subgraphs in structured KBs. To
bridge the gap between texts and structured KBs, we propose a Structured
Knowledge-aware Pre-training method (SKP). In the pre-training stage, we
introduce two novel structured knowledge-aware tasks, guiding the model to
effectively learn the implicit relationship and better representations of
complex subgraphs. In downstream KBQA task, we further design an efficient
linearization strategy and an interval attention mechanism, which assist the
model to better encode complex subgraphs and shield the interference of
irrelevant subgraphs during reasoning respectively. Detailed experiments and
analyses on WebQSP verify the effectiveness of SKP, especially the significant
improvement in subgraph retrieval (+4.08% H@10).