SETI: Systematicity Evaluation of Textual Inference
Xiyan Fu Anette Frank
We propose SETI (Systematicity Evaluation of Textual Inference), a novel and
comprehensive benchmark designed for evaluating pre-trained language models
(PLMs) for their systematicity capabilities in the domain of textual inference.
Specifically, SETI offers three different NLI tasks and corresponding datasets
to evaluate various types of systematicity in reasoning processes. In order to
solve these tasks, models are required to perform compositional inference based
on known primitive constituents. We conduct experiments of SETI on six widely
used PLMs. Results show that various PLMs are able to solve unseen
compositional inferences when having encountered the knowledge of how to
combine primitives, with good performance. However, they are considerably
limited when this knowledge is unknown to the model (40-100% points decrease).
Furthermore, we find that PLMs can improve drastically once exposed to crucial
compositional knowledge in minimalistic shots. These findings position SETI as
the first benchmark for measuring the future progress of PLMs in achieving
systematicity generalization in the textual inference.