Challenge Results Are Not Reproducible
Annika Reinke Georg Grab Lena Maier-Hein
While clinical trials are the state-of-the-art methods to assess the effect
of new medication in a comparative manner, benchmarking in the field of medical
image analysis is performed by so-called challenges. Recently, comprehensive
analysis of multiple biomedical image analysis challenges revealed large
discrepancies between the impact of challenges and quality control of the
design and reporting standard. This work aims to follow up on these results and
attempts to address the specific question of the reproducibility of the
participants methods. In an effort to determine whether alternative
interpretations of the method description may change the challenge ranking, we
reproduced the algorithms submitted to the 2019 Robust Medical Image
Segmentation Challenge (ROBUST-MIS). The leaderboard differed substantially
between the original challenge and reimplementation, indicating that challenge
rankings may not be sufficiently reproducible.