Analysis of Systems' Performance in Natural Language Processing
  Competitions
Sergio Nava-Muñoz Mario Graff Hugo Jair Escalante
Collaborative competitions have gained popularity in the scientific and
technological fields. These competitions involve defining tasks, selecting
evaluation scores, and devising result verification methods. In the standard
scenario, participants receive a training set and are expected to provide a
solution for a held-out dataset kept by organizers. An essential challenge for
organizers arises when comparing algorithms' performance, assessing multiple
participants, and ranking them. Statistical tools are often used for this
purpose; however, traditional statistical methods often fail to capture
decisive differences between systems' performance. This manuscript describes an
evaluation methodology for statistically analyzing competition results and
competition. The methodology is designed to be universally applicable; however,
it is illustrated using eight natural language competitions as case studies
involving classification and regression problems. The proposed methodology
offers several advantages, including off-the-shell comparisons with correction
mechanisms and the inclusion of confidence intervals. Furthermore, we introduce
metrics that allow organizers to assess the difficulty of competitions. Our
analysis shows the potential usefulness of our methodology for effectively
evaluating competition results.