Improving Language Models Meaning Understanding and Consistency by
  Learning Conceptual Roles from Dictionary
Myeongjun Erik Jang Thomas Lukasiewicz
The non-humanlike behaviour of contemporary pre-trained language models
(PLMs) is a leading cause undermining their trustworthiness. A striking
phenomenon of such faulty behaviours is the generation of inconsistent
predictions, which produces logically contradictory results, such as generating
different predictions for texts delivering the same meaning or violating
logical properties. Previous studies exploited data augmentation or implemented
specialised loss functions to alleviate the issue. However, their usage is
limited, because they consume expensive training resources for large-sized PLMs
and can only handle a certain consistency type. To this end, we propose a
practical approach that alleviates the inconsistent behaviour issue by
fundamentally improving PLMs' meaning awareness. Based on the conceptual role
theory, our method allows PLMs to capture accurate meaning by learning precise
interrelationships between concepts from word-definition pairs in a dictionary.
Next, we propose an efficient parameter integration technique that updates only
a few additional parameters to combine the learned interrelationship with PLMs'
pre-trained knowledge. Our experimental results reveal that the approach can
concurrently improve multiple types of consistency, enables efficient knowledge
integration, and easily applies to other languages.