Joint Pre-training and Local Re-training: Transferable Representation
  Learning on Multi-source Knowledge Graphs
Zequn Sun Jiacheng Huang Jinghao Lin Xiaozhou Xu Qijin Chen Wei Hu
In this paper, we present the ``joint pre-training and local re-training''
framework for learning and applying multi-source knowledge graph (KG)
embeddings. We are motivated by the fact that different KGs contain
complementary information to improve KG embeddings and downstream tasks. We
pre-train a large teacher KG embedding model over linked multi-source KGs and
distill knowledge to train a student model for a task-specific KG. To enable
knowledge transfer across different KGs, we use entity alignment to build a
linked subgraph for connecting the pre-trained KGs and the target KG. The
linked subgraph is re-trained for three-level knowledge distillation from the
teacher to the student, i.e., feature knowledge distillation, network knowledge
distillation, and prediction knowledge distillation, to generate more
expressive embeddings. The teacher model can be reused for different target KGs
and tasks without having to train from scratch. We conduct extensive
experiments to demonstrate the effectiveness and efficiency of our framework.