Ecologically Valid Explanations for Label Variation in NLI
Nan-Jiang Jiang Chenhao Tan Marie-Catherine de Marneffe
Human label variation, or annotation disagreement, exists in many natural
language processing (NLP) tasks, including natural language inference (NLI). To
gain direct evidence of how NLI label variation arises, we build LiveNLI, an
English dataset of 1,415 ecologically valid explanations (annotators explain
the NLI labels they chose) for 122 MNLI items (at least 10 explanations per
item). The LiveNLI explanations confirm that people can systematically vary on
their interpretation and highlight within-label variation: annotators sometimes
choose the same label for different reasons. This suggests that explanations
are crucial for navigating label interpretations in general. We few-shot prompt
large language models to generate explanations but the results are
inconsistent: they sometimes produces valid and informative explanations, but
it also generates implausible ones that do not support the label, highlighting
directions for improvement.