Dissecting Paraphrases: The Impact of Prompt Syntax and supplementary
  Information on Knowledge Retrieval from Pretrained Language Models
Stephan Linzbach Dimitar Dimitrov Laura Kallmeyer Kilian Evang Hajira Jabeen Stefan Dietze
Pre-trained Language Models (PLMs) are known to contain various kinds of
knowledge. One method to infer relational knowledge is through the use of
cloze-style prompts, where a model is tasked to predict missing subjects or
objects. Typically, designing these prompts is a tedious task because small
differences in syntax or semantics can have a substantial impact on knowledge
retrieval performance. Simultaneously, evaluating the impact of either prompt
syntax or information is challenging due to their interdependence. We designed
CONPARE-LAMA - a dedicated probe, consisting of 34 million distinct prompts
that facilitate comparison across minimal paraphrases. These paraphrases follow
a unified meta-template enabling the controlled variation of syntax and
semantics across arbitrary relations. CONPARE-LAMA enables insights into the
independent impact of either syntactical form or semantic information of
paraphrases on the knowledge retrieval performance of PLMs. Extensive knowledge
retrieval experiments using our probe reveal that prompts following clausal
syntax have several desirable properties in comparison to appositive syntax: i)
they are more useful when querying PLMs with a combination of supplementary
information, ii) knowledge is more consistently recalled across different
combinations of supplementary information, and iii) they decrease response
uncertainty when retrieving known facts. In addition, range information can
boost knowledge retrieval performance more than domain information, even though
domain information is more reliably helpful across syntactic forms.