Expanding the Vocabulary of BERT for Knowledge Base Construction
Dong Yang Xu Wang Remzi Celebi
Knowledge base construction entails acquiring structured information to
create a knowledge base of factual and relational data, facilitating question
answering, information retrieval, and semantic understanding. The challenge
called "Knowledge Base Construction from Pretrained Language Models" at
International Semantic Web Conference 2023 defines tasks focused on
constructing knowledge base using language model. Our focus was on Track 1 of
the challenge, where the parameters are constrained to a maximum of 1 billion,
and the inclusion of entity descriptions within the prompt is prohibited.
  Although the masked language model offers sufficient flexibility to extend
its vocabulary, it is not inherently designed for multi-token prediction. To
address this, we present Vocabulary Expandable BERT for knowledge base
construction, which expand the language model's vocabulary while preserving
semantic embeddings for newly added words. We adopt task-specific
re-pre-training on masked language model to further enhance the language model.
  Through experimentation, the results show the effectiveness of our
approaches. Our framework achieves F1 score of 0.323 on the hidden test set and
0.362 on the validation set, both data set is provided by the challenge.
Notably, our framework adopts a lightweight language model (BERT-base, 0.13
billion parameters) and surpasses the model using prompts directly on large
language model (Chatgpt-3, 175 billion parameters). Besides, Token-Recode
achieves comparable performances as Re-pretrain. This research advances
language understanding models by enabling the direct embedding of multi-token
entities, signifying a substantial step forward in link prediction task in
knowledge graph and metadata completion in data management.