What Do LLMs Associate with Your Name? A Human-Centered Black-Box Audit of Personal Data
This work addresses privacy concerns for users interacting with LLMs, highlighting a need for control over model-generated associations, though it is incremental in building on prior audits of personal data resurfacing.
The paper tackled the problem of understanding how strongly large language models (LLMs) associate personal data with individuals' names, showing that models like GPT-4o can generate 11 personal features with 60% or more accuracy for everyday users.
Large language models (LLMs), and conversational agents based on them, are exposed to personal data (PD) during pre-training and during user interactions. Prior work shows that PD can resurface, yet users lack insight into how strongly models associate specific information to their identity. We audit PD across eight LLMs (3 open-source; 5 API-based, including GPT-4o), introduce LMP2 (Language Model Privacy Probe), a human-centered, privacy-preserving audit tool refined through two formative studies (N=20), and run two studies with EU residents to capture (i) intuitions about LLM-generated PD (N1=155) and (ii) reactions to tool output (N2=303). We show empirically that models confidently generate multiple PD categories for well-known individuals. For everyday users, GPT-4o generates 11 features with 60% or more accuracy (e.g., gender, hair color, languages). Finally, 72% of participants sought control over model-generated associations with their name, raising questions about what counts as PD and whether data privacy rights should extend to LLMs.