Do PLMs Know and Understand Ontological Knowledge?
Weiqi Wu Chengyue Jiang Yong Jiang Pengjun Xie Kewei Tu
Ontological knowledge, which comprises classes and properties and their
relationships, is integral to world knowledge. It is significant to explore
whether Pretrained Language Models (PLMs) know and understand such knowledge.
However, existing PLM-probing studies focus mainly on factual knowledge,
lacking a systematic probing of ontological knowledge. In this paper, we focus
on probing whether PLMs store ontological knowledge and have a semantic
understanding of the knowledge rather than rote memorization of the surface
form. To probe whether PLMs know ontological knowledge, we investigate how well
PLMs memorize: (1) types of entities; (2) hierarchical relationships among
classes and properties, e.g., Person is a subclass of Animal and Member of
Sports Team is a subproperty of Member of ; (3) domain and range constraints of
properties, e.g., the subject of Member of Sports Team should be a Person and
the object should be a Sports Team. To further probe whether PLMs truly
understand ontological knowledge beyond memorization, we comprehensively study
whether they can reliably perform logical reasoning with given knowledge
according to ontological entailment rules. Our probing results show that PLMs
can memorize certain ontological knowledge and utilize implicit knowledge in
reasoning. However, both the memorizing and reasoning performances are less
than perfect, indicating incomplete knowledge and understanding.