Evaluating Large Language Models Using Contrast Sets: An Experimental
  Approach
Manish Sanwal
In the domain of Natural Language Inference (NLI), especially in tasks
involving the classification of multiple input texts, the Cross-Entropy Loss
metric is widely employed as a standard for error measurement. However, this
metric falls short in effectively evaluating a model's capacity to understand
language entailments. In this study, we introduce an innovative technique for
generating a contrast set for the Stanford Natural Language Inference (SNLI)
dataset. Our strategy involves the automated substitution of verbs, adverbs,
and adjectives with their synonyms to preserve the original meaning of
sentences. This method aims to assess whether a model's performance is based on
genuine language comprehension or simply on pattern recognition. We conducted
our analysis using the ELECTRA-small model. The model achieved an accuracy of
89.9% on the conventional SNLI dataset but showed a reduced accuracy of 72.5%
on our contrast set, indicating a substantial 17% decline. This outcome led us
to conduct a detailed examination of the model's learning behaviors. Following
this, we improved the model's resilience by fine-tuning it with a
contrast-enhanced training dataset specifically designed for SNLI, which
increased its accuracy to 85.5% on the contrast sets. Our findings highlight
the importance of incorporating diverse linguistic expressions into datasets
for NLI tasks. We hope that our research will encourage the creation of more
inclusive datasets, thereby contributing to the development of NLI models that
are both more sophisticated and effective.