Distributed Evaluations: Ending Neural Point Metrics
This addresses reproducibility issues for researchers and practitioners using neural models in IR, especially in low-resource settings, but is incremental as it builds on existing evaluation practices.
The paper tackles the problem of unreliable evaluation of neural models in information retrieval due to randomness and hyperparameter sensitivity, proposing a method to evaluate models over multiple random seeds and hyperparameter configurations to improve reproducibility and robustness.
With the rise of neural models across the field of information retrieval, numerous publications have incrementally pushed the envelope of performance for a multitude of IR tasks. However, these networks often sample data in random order, are initialized randomly, and their success is determined by a single evaluation score. These issues are aggravated by neural models achieving incremental improvements from previous neural baselines, leading to multiple near state of the art models that are difficult to reproduce and quickly become deprecated. As neural methods are starting to be incorporated into low resource and noisy collections that further exacerbate this issue, we propose evaluating neural models both over multiple random seeds and a set of hyperparameters within $ε$ distance of the chosen configuration for a given metric.