A Distributed and Accountable Approach to Offline Recommender Systems Evaluation
This addresses the issue of inconsistent experimental protocols and lack of transparency in recommender system evaluations for researchers and practitioners, though it is incremental as it builds on existing web technologies.
The authors tackled the problem of incomparable and non-accountable offline evaluations in recommender systems by introducing RecLab, an open-source software that enables distributed evaluation through RESTful APIs and a web interface, resulting in a tool that computes and publicly stores comprehensive metrics for easy integration and analysis.
Different software tools have been developed with the purpose of performing offline evaluations of recommender systems. However, the results obtained with these tools may be not directly comparable because of subtle differences in the experimental protocols and metrics. Furthermore, it is difficult to analyze in the same experimental conditions several algorithms without disclosing their implementation details. For these reasons, we introduce RecLab, an open source software for evaluating recommender systems in a distributed fashion. By relying on consolidated web protocols, we created RESTful APIs for training and querying recommenders remotely. In this way, it is possible to easily integrate into the same toolkit algorithms realized with different technologies. In details, the experimenter can perform an evaluation by simply visiting a web interface provided by RecLab. The framework will then interact with all the selected recommenders and it will compute and display a comprehensive set of measures, each representing a different metric. The results of all experiments are permanently stored and publicly available in order to support accountability and comparative analyses.