Pre-training and Diagnosing Knowledge Base Completion Models
Vid Kocijan Myeongjun Erik Jang Thomas Lukasiewicz
In this work, we introduce and analyze an approach to knowledge transfer from
one collection of facts to another without the need for entity or relation
matching. The method works for both canonicalized knowledge bases and
uncanonicalized or open knowledge bases, i.e., knowledge bases where more than
one copy of a real-world entity or relation may exist. The main contribution is
a method that can make use of large-scale pre-training on facts, which were
collected from unstructured text, to improve predictions on structured data
from a specific domain. The introduced method is most impactful on small
datasets such as ReVerb20k, where a 6% absolute increase of mean reciprocal
rank and 65% relative decrease of mean rank over the previously best method was
achieved, despite not relying on large pre-trained models like Bert. To
understand the obtained pre-trained models better, we then introduce a novel
dataset for the analysis of pre-trained models for Open Knowledge Base
Completion, called Doge (Diagnostics of Open knowledge Graph Embeddings). It
consists of 6 subsets and is designed to measure multiple properties of a
pre-trained model: robustness against synonyms, ability to perform deductive
reasoning, presence of gender stereotypes, consistency with reverse relations,
and coverage of different areas of general knowledge. Using the introduced
dataset, we show that the existing OKBC models lack consistency in the presence
of synonyms and inverse relations and are unable to perform deductive
reasoning. Moreover, their predictions often align with gender stereotypes,
which persist even when presented with counterevidence. We additionally
investigate the role of pre-trained word embeddings and demonstrate that
avoiding biased word embeddings is not a sufficient measure to prevent biased
behavior of OKBC models.