CLIFT: Analysing Natural Distribution Shift on Question Answering Models
  in Clinical Domain
Ankit Pal
This paper introduces a new testbed CLIFT (Clinical Shift) for the clinical
domain Question-answering task. The testbed includes 7.5k high-quality question
answering samples to provide a diverse and reliable benchmark. We performed a
comprehensive experimental study and evaluated several QA deep-learning models
under the proposed testbed. Despite impressive results on the original test
set, the performance degrades when applied to new test sets, which shows the
distribution shift. Our findings emphasize the need for and the potential for
increasing the robustness of clinical domain models under distributional
shifts. The testbed offers one way to track progress in that direction. It also
highlights the necessity of adopting evaluation metrics that consider
robustness to natural distribution shifts. We plan to expand the corpus by
adding more samples and model results. The full paper and the updated benchmark
are available at github.com/openlifescience-ai/clift