Beyond the Answers: Reviewing the Rationality of Multiple Choice
  Question Answering for the Evaluation of Large Language Models
Haochun Wang Sendong Zhao Zewen Qiang Bing Qin Ting Liu
In the field of natural language processing (NLP), Large Language Models
(LLMs) have precipitated a paradigm shift, markedly enhancing performance in
natural language generation tasks. Despite these advancements, the
comprehensive evaluation of LLMs remains an inevitable challenge for the
community. Recently, the utilization of Multiple Choice Question Answering
(MCQA) as a benchmark for LLMs has gained considerable traction. This study
investigates the rationality of MCQA as an evaluation method for LLMs. If LLMs
genuinely understand the semantics of questions, their performance should
exhibit consistency across the varied configurations derived from the same
questions. Contrary to this expectation, our empirical findings suggest a
notable disparity in the consistency of LLM responses, which we define as
REsponse VAriability Syndrome (REVAS) of the LLMs, indicating that current
MCQA-based benchmarks may not adequately capture the true capabilities of LLMs,
which underscores the need for more robust evaluation mechanisms in assessing
the performance of LLMs.