Self-Distillation with Meta Learning for Knowledge Graph Completion
Yunshui Li Junhao Liu Chengming Li Min Yang
In this paper, we propose a selfdistillation framework with meta
learning(MetaSD) for knowledge graph completion with dynamic pruning, which
aims to learn compressed graph embeddings and tackle the longtail samples.
Specifically, we first propose a dynamic pruning technique to obtain a small
pruned model from a large source model, where the pruning mask of the pruned
model could be updated adaptively per epoch after the model weights are
updated. The pruned model is supposed to be more sensitive to difficult to
memorize samples(e.g., longtail samples) than the source model. Then, we
propose a onestep meta selfdistillation method for distilling comprehensive
knowledge from the source model to the pruned model, where the two models
coevolve in a dynamic manner during training. In particular, we exploit the
performance of the pruned model, which is trained alongside the source model in
one iteration, to improve the source models knowledge transfer ability for the
next iteration via meta learning. Extensive experiments show that MetaSD
achieves competitive performance compared to strong baselines, while being 10x
smaller than baselines.