Testing the Consistency of Performance Scores Reported for Binary
  Classification Problems
Attila Fazekas György Kovács
Binary classification is a fundamental task in machine learning, with
applications spanning various scientific domains. Whether scientists are
conducting fundamental research or refining practical applications, they
typically assess and rank classification techniques based on performance
metrics such as accuracy, sensitivity, and specificity. However, reported
performance scores may not always serve as a reliable basis for research
ranking. This can be attributed to undisclosed or unconventional practices
related to cross-validation, typographical errors, and other factors. In a
given experimental setup, with a specific number of positive and negative test
items, most performance scores can assume specific, interrelated values. In
this paper, we introduce numerical techniques to assess the consistency of
reported performance scores and the assumed experimental setup. Importantly,
the proposed approach does not rely on statistical inference but uses numerical
methods to identify inconsistencies with certainty. Through three different
applications related to medicine, we demonstrate how the proposed techniques
can effectively detect inconsistencies, thereby safeguarding the integrity of
research fields. To benefit the scientific community, we have made the
consistency tests available in an open-source Python package.