Multi-Set Inoculation: Assessing Model Robustness Across Multiple
  Challenge Sets
Vatsal Gupta Pranshu Pandya Tushar Kataria Vivek Gupta Dan Roth
Language models, given their black-box nature, often exhibit sensitivity to
input perturbations, leading to trust issues due to hallucinations. To bolster
trust, it's essential to understand these models' failure modes and devise
strategies to enhance their performance. In this study, we propose a framework
to study the effect of input perturbations on language models of different
scales, from pre-trained models to large language models (LLMs). We use
fine-tuning to train a robust model to perturbations, and we investigate
whether exposure to one perturbation improves or degrades the model's
performance on other perturbations. To address multi-perturbation robustness,
we suggest three distinct training strategies. We also extend the framework to
LLMs via a chain of thought(COT) prompting with exemplars. We instantiate our
framework for the Tabular-NLI task and show that the proposed strategies train
the model robust to different perturbations without losing accuracy on a given
dataset.