Jamp: Controlled Japanese Temporal Inference Dataset for Evaluating
  Generalization Capacity of Language Models
Tomoki Sugimoto Yasumasa Onoe Hitomi Yanaka
Natural Language Inference (NLI) tasks involving temporal inference remain
challenging for pre-trained language models (LMs). Although various datasets
have been created for this task, they primarily focus on English and do not
address the need for resources in other languages. It is unclear whether
current LMs realize the generalization capacity for temporal inference across
languages. In this paper, we present Jamp, a Japanese NLI benchmark focused on
temporal inference. Our dataset includes a range of temporal inference
patterns, which enables us to conduct fine-grained analysis. To begin the data
annotation process, we create diverse inference templates based on the formal
semantics test suites. We then automatically generate diverse NLI examples by
using the Japanese case frame dictionary and well-designed templates while
controlling the distribution of inference patterns and gold labels. We evaluate
the generalization capacities of monolingual/multilingual LMs by splitting our
dataset based on tense fragments (i.e., temporal inference patterns). Our
findings demonstrate that LMs struggle with specific linguistic phenomena, such
as habituality, indicating that there is potential for the development of more
effective NLI models across languages.