Streamlining Evaluation with ir-measures
This tool addresses the problem of cumbersome evaluation for information retrieval researchers, though it is incremental as it builds on existing tools rather than creating new methods.
The paper introduces ir-measures, a tool that simplifies evaluation in information retrieval by providing a common interface to calculate diverse metrics, including recent ones like those from the C/W/L framework, automatically invoking necessary tools to streamline the process.
We present ir-measures, a new tool that makes it convenient to calculate a diverse set of evaluation measures used in information retrieval. Rather than implementing its own measure calculations, ir-measures provides a common interface to a handful of evaluation tools. The necessary tools are automatically invoked (potentially multiple times) to calculate all the desired metrics, simplifying the evaluation process for the user. The tool also makes it easier for researchers to use recently-proposed measures (such as those from the C/W/L framework) alongside traditional measures, potentially encouraging their adoption.