Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A
  Multifaceted Statistical Approach
Kun Sun Rong Wang Haitao Liu Anders Søgaard
Amidst the rapid evolution of LLMs, the significance of evaluation in
comprehending and propelling these models forward is increasingly paramount.
Evaluations have revealed that factors such as scaling, training types,
architectures and other factors profoundly impact the performance of LLMs.
However, the extent and nature of these impacts continue to be subjects of
debate because most assessments have been restricted to a limited number of
models and data points. Clarifying the effects of these factors on performance
scores can be more effectively achieved through a statistical lens. Our study
embarks on a thorough re-examination of these LLMs, targeting the inadequacies
in current evaluation methods. With the advent of a uniform evaluation
framework, our research leverages an expansive dataset of evaluation results,
introducing a comprehensive statistical methodology. This includes the
application of ANOVA, Tukey HSD tests, GAMM, and clustering technique, offering
a robust and transparent approach to deciphering LLM performance data. Contrary
to prevailing findings, our results challenge assumptions about emergent
abilities and the influence of given training types and architectures in LLMs.
These findings furnish new perspectives on the characteristics, intrinsic
nature, and developmental trajectories of LLMs. By providing straightforward
and reliable methods to scrutinize and reassess LLM performance data, this
study contributes a nuanced perspective on LLM efficiency and potentials.