Ranger: A Toolkit for Effect-Size Based Multi-Task Evaluation
Mete Sertkan Sophia Althammer Sebastian Hofstätter
In this paper, we introduce Ranger - a toolkit to facilitate the easy use of
effect-size-based meta-analysis for multi-task evaluation in NLP and IR. We
observed that our communities often face the challenge of aggregating results
over incomparable metrics and scenarios, which makes conclusions and take-away
messages less reliable. With Ranger, we aim to address this issue by providing
a task-agnostic toolkit that combines the effect of a treatment on multiple
tasks into one statistical evaluation, allowing for comparison of metrics and
computation of an overall summary effect. Our toolkit produces
publication-ready forest plots that enable clear communication of evaluation
results over multiple tasks. Our goal with the ready-to-use Ranger toolkit is
to promote robust, effect-size-based evaluation and improve evaluation
standards in the community. We provide two case studies for common IR and NLP
settings to highlight Ranger's benefits.