Sometimes You Want to Go Where Everybody Knows your Name
This work addresses the challenge of personalizing models for users with limited data and privacy constraints, though it is incremental as it primarily introduces a metric and a simple example.
The paper tackles the problem of measuring model personalization to individual user preferences while balancing performance on user-specific data and a global dataset, introducing a new metric and demonstrating its use in a sentiment classification simulation with distinct user vocabularies.
We introduce a new metric for measuring how well a model personalizes to a user's specific preferences. We define personalization as a weighting between performance on user specific data and performance on a more general global dataset that represents many different users. This global term serves as a form of regularization that forces us to not overfit to individual users who have small amounts of data. In order to protect user privacy, we add the constraint that we may not centralize or share user data. We also contribute a simple experiment in which we simulate classifying sentiment for users with very distinct vocabularies. This experiment functions as an example of the tension between doing well globally on all users, and doing well on any specific individual user. It also provides a concrete example of how to employ our new metric to help reason about and resolve this tension. We hope this work can help frame and ground future work into personalization.