Jury: A Comprehensive Evaluation Toolkit
Devrim Cavusoglu Ulas Sert Secil Sen Sinan Altinuc
Evaluation plays a critical role in deep learning as a fundamental block of
any prediction-based system. However, the vast number of Natural Language
Processing (NLP) tasks and the development of various metrics have led to
challenges in evaluating different systems with different metrics. To address
these challenges, we introduce jury, a toolkit that provides a unified
evaluation framework with standardized structures for performing evaluation
across different tasks and metrics. The objective of jury is to standardize and
improve metric evaluation for all systems and aid the community in overcoming
the challenges in evaluation. Since its open-source release, jury has reached a
wide audience and is available at https://github.com/obss/jury.