From Optimizing Engagement to Measuring Value
This addresses the gap between engagement and value in recommendation systems for designers and platforms, offering a novel approach but with incremental improvements in measurement.
The paper tackles the problem that recommendation engines often optimize for user engagement (e.g., clicks) rather than a desired notion of 'value', and proposes a measurement theory framework to operationalize and optimize for value, implementing it on Twitter with millions of users and evaluating its validity qualitatively.
Most recommendation engines today are based on predicting user engagement, e.g. predicting whether a user will click on an item or not. However, there is potentially a large gap between engagement signals and a desired notion of "value" that is worth optimizing for. We use the framework of measurement theory to (a) confront the designer with a normative question about what the designer values, (b) provide a general latent variable model approach that can be used to operationalize the target construct and directly optimize for it, and (c) guide the designer in evaluating and revising their operationalization. We implement our approach on the Twitter platform on millions of users. In line with established approaches to assessing the validity of measurements, we perform a qualitative evaluation of how well our model captures a desired notion of "value".