Anay Agarwalla, Simeon Sayer
Sensitive attributes are legally protected characteristics that should not be used to discriminate. Careful steps have been taken to minimize the risk of human bias regarding these fields, such as race and age. Large language models (LLMs) are similarly trained not to attempt to infer these aspects. However, just because they shouldn't, doesn't mean they don't. Using chat-like text fragments from authors tagged with sensitive attributes (e.g., MBTI personality, country of origin, gender), a model can often classify these attributes better than a naive guess, with results depending on the combination of subject matter and attribute. The text data from these comments is converted into numerical representations using embedding models, which are then used to train relatively simple classifiers such as logistic regression and decision trees. This study's results show that even these lightweight models can detect statistically significant signals associated with sensitive attributes in user-generated text. The results show that demographic traits such as gender and age are more readily predictable, whereas personality traits are expressed more subtly and depend more heavily on context. Predictive performance varies across online Reddit communities, with some subreddits consistently revealing attributes, while others show high variability depending on the trait being analyzed. These findings indicate that language contains latent identity signals that users may not intend to disclose but are nevertheless detectable through computational methods, and imply that more complex language models may have an inherent, greater capacity to infer sensitive attributes. This raises important concerns about privacy, bias, and the potential misuse of inferred personal information in AI systems. We call for increased transparency, stronger safeguards, and careful policy consideration for future LLMs.