Debiasing Large Language Models toward Social Factors in Online Behavior Analytics through Prompt Knowledge Tuning
For researchers and practitioners using LLMs in social media analysis, this work addresses a specific bias (social attribution) that can skew behavior analytics, but the method is domain-specific and incremental.
The paper investigates how Large Language Models (LLMs) exhibit social-attribution bias in behavior analytics and proposes a prompt-tuning method that incorporates user goals and message context to reduce this bias. Experiments on intent and theme detection in disaster-related social media show improved performance and reduced bias across Llama3, Mistral, and Gemma.
Attribution theory explains how individuals interpret and attribute others' behavior in a social context by employing personal (dispositional) and impersonal (situational) causality. Large Language Models (LLMs), trained on human-generated corpora, may implicitly mimic this social attribution process in social contexts. However, the extent to which LLMs utilize these causal attributions in their reasoning remains underexplored. Although using reasoning paradigms, such as Chain-of-Thought (CoT), has shown promising results in various tasks, ignoring social attribution in reasoning could lead to biased responses by LLMs in social contexts. In this study, we investigate the impact of incorporating a user's goal as knowledge to infer dispositional causality and message context to infer situational causality on LLM performance. To this end, we introduce a scalable method to mitigate such biases by enriching the instruction prompts for LLMs with two prompt aids using social-attribution knowledge, based on the context and goal of a social media message. This method improves the model performance while reducing the social-attribution bias of the LLM in the reasoning on zero-shot classification tasks for behavior analytics applications. We empirically show the benefits of our method across two tasks-intent detection and theme detection on social media in the disaster domain-when considering the variability of disaster types and multiple languages of social media. Our experiments highlight the biases of three open-source LLMs: Llama3, Mistral, and Gemma, toward social attribution, and show the effectiveness of our mitigation strategies.