ConcEPT: Concept-Enhanced Pre-Training for Language Models
Xintao Wang Zhouhong Gu Jiaqing Liang Dakuan Lu Yanghua Xiao Wei Wang
Pre-trained language models (PLMs) have been prevailing in state-of-the-art
methods for natural language processing, and knowledge-enhanced PLMs are
further proposed to promote model performance in knowledge-intensive tasks.
However, conceptual knowledge, one essential kind of knowledge for human
cognition, still remains understudied in this line of research. This limits
PLMs' performance in scenarios requiring human-like cognition, such as
understanding long-tail entities with concepts. In this paper, we propose
ConcEPT, which stands for Concept-Enhanced Pre-Training for language models, to
infuse conceptual knowledge into PLMs. ConcEPT exploits external taxonomies
with entity concept prediction, a novel pre-training objective to predict the
concepts of entities mentioned in the pre-training contexts. Unlike previous
concept-enhanced methods, ConcEPT can be readily adapted to various downstream
applications without entity linking or concept mapping. Results of extensive
experiments show the effectiveness of ConcEPT in four tasks such as entity
typing, which validates that our model gains improved conceptual knowledge with
concept-enhanced pre-training.