Offline Evaluation of Reward-Optimizing Recommender Systems: The Case of Simulation
This work tackles the problem of costly and unreliable offline evaluation for researchers and practitioners in recommender systems, proposing a shift towards simulation as a more practical approach, though it appears incremental in nature.
The paper addresses the challenge of offline evaluation for reward-optimizing recommender systems, arguing that simulation-based comparisons offer a more reliable alternative to existing proxy-based and counterfactual methods, which often lack correlation with online metrics or rely on unrealistic assumptions.
Both in academic and industry-based research, online evaluation methods are seen as the golden standard for interactive applications like recommendation systems. Naturally, the reason for this is that we can directly measure utility metrics that rely on interventions, being the recommendations that are being shown to users. Nevertheless, online evaluation methods are costly for a number of reasons, and a clear need remains for reliable offline evaluation procedures. In industry, offline metrics are often used as a first-line evaluation to generate promising candidate models to evaluate online. In academic work, limited access to online systems makes offline metrics the de facto approach to validating novel methods. Two classes of offline metrics exist: proxy-based methods, and counterfactual methods. The first class is often poorly correlated with the online metrics we care about, and the latter class only provides theoretical guarantees under assumptions that cannot be fulfilled in real-world environments. Here, we make the case that simulation-based comparisons provide ways forward beyond offline metrics, and argue that they are a preferable means of evaluation.