Amir Goldberg

Caleb Ziems, William Held, Jane Yu et al.

To serve global users safely and productively, LLMs need culture-specific knowledge that might not be learned during pre-training. How do we find such knowledge that is (1) salient to in-group users, but (2) unknown to LLMs? The most common solutions are single-initiative: either researchers define challenging questions that users passively answer (traditional annotation), or users actively produce data that researchers structure as benchmarks (knowledge extraction). The process would benefit from mixed-initiative collaboration, where users guide the process to meaningfully reflect their cultures, and LLMs steer the process towards more challenging questions that meet the researcher's goals. We propose a mixed-initiative methodology called CultureCartography. Here, an LLM initializes annotation with questions for which it has low-confidence answers, making explicit both its prior knowledge and the gaps therein. This allows a human respondent to fill these gaps and steer the model towards salient topics through direct edits. We implement this methodology as a tool called CultureExplorer. Compared to a baseline where humans answer LLM-proposed questions, we find that CultureExplorer more effectively produces knowledge that leading models like DeepSeek R1 and GPT-4o are missing, even with web search. Fine-tuning on this data boosts the accuracy of Llama-3.1-8B by up to 19.2% on related culture benchmarks.

Xinrui Chloe Zhao, Douglas Guilbeault, Amir Goldberg

Recent studies argue that LLMs can predict human stereotypical judgments. Yet whether LLMs emulate the cognitive processes underlying human stereotypes, or merely retrieve learned associations to solve prediction tasks, remains unclear. Prior work examines LLMs' stereotypes in either (i) controlled judgment tasks like multiple choice surveys, or (ii) contexts constrained by conventionalized and predictable group biases. Here, we compare the structure of the stereotypes that humans and LLMs exhibit in the interpretation of free-form stimuli, namely abstract art and Rorschach blots, which lack pre-established cultural meanings. We recruit participants across five social domains (gender, partisanship, personality, urbanicity, and lifestyle) and elicit both first-order (direct personal interpretations) and second-order responses (predictions about how members of social groups will interpret the stimuli); we replicate this design with two multimodal models (GPT-4o mini and Llama-3.2-11B-Vision-Instruct). Humans and LLMs differ not only in magnitude but in the qualitative nature of their stereotypes. Human first-order responses display heterogeneity with minimal group structure. When predicting group responses, humans engage in "stereotype exaggeration" by moderately amplifying first-order tendencies while preserving diversity. By contrast, LLMs exhibit homogeneous first-order responses, and yet generate stark second-order stereotypes that neither amplify existing first-order tendencies nor reflect actual human group differences, a process we term "stereotype hallucination." LLMs continued to hallucinate stereotypes even when fine-tuned on the response data of actual participants. These findings suggest significant limitations in the use of LLMs to model and predict human behavior in novel contexts involving diverse interpretations.