Overlearning Reveals Sensitive Attributes
This reveals a critical privacy and bias vulnerability in machine learning models, affecting users and practitioners by exposing sensitive information despite safeguards.
The paper tackles the problem of 'overlearning', where models trained for simple tasks implicitly learn sensitive attributes not part of the objective, such as race or identity, even without training data for those attributes. They demonstrate this in vision and NLP models, showing it breaks privacy protections and enables re-purposing for privacy-violating tasks, with analysis indicating it is intrinsic and cannot be prevented by censoring.
"Overlearning" means that a model trained for a seemingly simple objective implicitly learns to recognize attributes and concepts that are (1) not part of the learning objective, and (2) sensitive from a privacy or bias perspective. For example, a binary gender classifier of facial images also learns to recognize races\textemdash even races that are not represented in the training data\textemdash and identities. We demonstrate overlearning in several vision and NLP models and analyze its harmful consequences. First, inference-time representations of an overlearned model reveal sensitive attributes of the input, breaking privacy protections such as model partitioning. Second, an overlearned model can be "re-purposed" for a different, privacy-violating task even in the absence of the original training data. We show that overlearning is intrinsic for some tasks and cannot be prevented by censoring unwanted attributes. Finally, we investigate where, when, and why overlearning happens during model training.