Online Evaluations for Everyone: Mr. DLib's Living Lab for Scholarly Recommendations
This provides a practical tool for researchers to test algorithms in real-world settings, though it is incremental as it builds on existing service infrastructure.
The paper tackles the lack of online evaluation platforms for scholarly recommender systems by introducing Mr. DLib's living lab, which delivered 1,826,643 recommendations with a 0.21% average click-through rate over sixteen months.
We introduce the first 'living lab' for scholarly recommender systems. This lab allows recommender-system researchers to conduct online evaluations of their novel algorithms for scholarly recommendations, i.e., recommendations for research papers, citations, conferences, research grants, etc. Recommendations are delivered through the living lab's API to platforms such as reference management software and digital libraries. The living lab is built on top of the recommender-system as-a-service Mr. DLib. Current partners are the reference management software JabRef and the CORE research team. We present the architecture of Mr. DLib's living lab as well as usage statistics on the first sixteen months of operating it. During this time, 1,826,643 recommendations were delivered with an average click-through rate of 0.21%.