Jury: A Comprehensive Evaluation Toolkit
This toolkit addresses evaluation standardization problems for the NLP community, but it is incremental as it builds on existing metrics and frameworks.
The authors tackled the challenge of evaluating diverse NLP systems with varied metrics by introducing jury, a toolkit that provides a unified evaluation framework, which has gained wide adoption since its open-source release.
Evaluation plays a critical role in deep learning as a fundamental block of any prediction-based system. However, the vast number of Natural Language Processing (NLP) tasks and the development of various metrics have led to challenges in evaluating different systems with different metrics. To address these challenges, we introduce jury, a toolkit that provides a unified evaluation framework with standardized structures for performing evaluation across different tasks and metrics. The objective of jury is to standardize and improve metric evaluation for all systems and aid the community in overcoming the challenges in evaluation. Since its open-source release, jury has reached a wide audience and is available at https://github.com/obss/jury.