When Personalization Harms: Reconsidering the Use of Group Attributes in Prediction
This addresses fairness and performance issues in personalized prediction for groups, particularly in clinical settings, but is incremental as it builds on existing fairness and personalization concepts.
The paper tackles the problem of machine learning models that use group attributes for personalization, showing they can reduce performance at the group level, and proposes formal conditions for 'fair use' to ensure tailored performance gains, with empirical results demonstrating prevalent violations and simple mitigation interventions.
Machine learning models are often personalized with categorical attributes that are protected, sensitive, self-reported, or costly to acquire. In this work, we show models that are personalized with group attributes can reduce performance at a group level. We propose formal conditions to ensure the "fair use" of group attributes in prediction tasks by training one additional model -- i.e., collective preference guarantees to ensure that each group who provides personal data will receive a tailored gain in performance in return. We present sufficient conditions to ensure fair use in empirical risk minimization and characterize failure modes that lead to fair use violations due to standard practices in model development and deployment. We present a comprehensive empirical study of fair use in clinical prediction tasks. Our results demonstrate the prevalence of fair use violations in practice and illustrate simple interventions to mitigate their harm.