Pytrec_eval: An Extremely Fast Python Interface to trec_eval
This provides a performance improvement for researchers and practitioners in information retrieval who need efficient evaluation tools, but it is incremental as it builds on existing trec_eval.
The authors tackled the problem of slow evaluation in information retrieval by introducing pytrec_eval, a fast Python interface to trec_eval, which is about 10 times faster than subprocess calls and twice as fast as native Python implementations for typical rankings.
We introduce pytrec_eval, a Python interface to the tree_eval information retrieval evaluation toolkit. pytrec_eval exposes the reference implementations of trec_eval within Python as a native extension. We show that pytrec_eval is around one order of magnitude faster than invoking trec_eval as a sub process from within Python. Compared to a native Python implementation of NDCG, pytrec_eval is twice as fast for practically-sized rankings. Finally, we demonstrate its effectiveness in an application where pytrec_eval is combined with Pyndri and the OpenAI Gym where query expansion is learned using Q-learning.