Leveraging Multi-Method Evaluation for Multi-Stakeholder Settings
This addresses the challenge of fair and comprehensive evaluation in multi-stakeholder recommendation settings, though it appears incremental as it builds on existing evaluation concepts.
The paper tackles the problem of evaluating recommendation systems with multiple stakeholders by proposing a multi-method evaluation approach, which combines multiple evaluation methods to provide a richer assessment and prevent blind spots.
In this paper, we focus on recommendation settings with multiple stakeholders with possibly varying goals and interests, and argue that a single evaluation method or measure is not able to evaluate all relevant aspects in such a complex setting. We reason that employing a multi-method evaluation, where multiple evaluation methods or measures are combined and integrated, allows for getting a richer picture and prevents blind spots in the evaluation outcome.