QADYNAMICS: Training Dynamics-Driven Synthetic QA Diagnostic for
  Zero-Shot Commonsense Question Answering
Haochen Shi Weiqi Wang Tianqing Fang Baixuan Xu Wenxuan Ding Xin Liu Yangqiu Song
Zero-shot commonsense Question-Answering (QA) requires models to reason about
general situations beyond specific benchmarks. State-of-the-art approaches
fine-tune language models on QA pairs constructed from CommonSense Knowledge
Bases (CSKBs) to equip the models with more commonsense knowledge in a QA
context. However, current QA synthesis protocols may introduce noise from the
CSKBs and generate ungrammatical questions and false negative options, which
impede the model's ability to generalize. To address these issues, we propose
QADYNAMICS, a training dynamics-driven framework for QA diagnostics and
refinement. Our approach analyzes the training dynamics of each QA pair at both
the question level and option level, discarding machine-detectable artifacts by
removing uninformative QA pairs and mislabeled or false-negative options.
Extensive experiments demonstrate the effectiveness of our approach, which
outperforms all baselines while using only 33% of the synthetic data, even
including LLMs such as ChatGPT. Moreover, expert evaluations confirm that our
framework significantly improves the quality of QA synthesis. Our codes and
model checkpoints are available at
https://github.com/HKUST-KnowComp/QaDynamics.