Contextualized Prompting For Stance Detection On Social Media
For researchers and practitioners using LLMs for stance detection on social media, this work provides nuanced insights into when and how context helps or hurts, but the findings are incremental and dataset-specific.
The paper investigates how incorporating contextual features (e.g., user biographies, LLM-generated target descriptions) into zero-shot prompting affects stance detection on Twitter, finding that LLM-generated target descriptions consistently improve accuracy, while other context can be detrimental. Across four datasets, they show that including other tweets by the same user can impair performance due to noise.
Stance detection on social media is challenging due to short, noisy, and context-dependent language. While large language models (LLMs) show zero-shot generalization, they are typically prompted without contextual information, which limits their ability to interpret ambiguous posts. In this work, we systematically investigate the impact of incorporating real-world (e.g., user biographies), derived (e.g., political party), and LLM-generated (e.g., target descriptions) contextual features into zero-shot prompting for stance detection on Twitter. Our evaluation spans four benchmark datasets, including a new high-quality German Twitter stance dataset. Across multiple LLMs, we find that integrating contextual information improves performance, but only under specific conditions. LLM-generated target descriptions consistently enhance accuracy, while other user metadata has mixed or even detrimental effects. Notably, we show that the inclusion of other tweets by the same user, often beneficial in supervised learning, can impair performance due to input noise. Our qualitative analysis reveals that LLMs struggle to distinguish task-specific useful information from irrelevant context. Our findings highlight both the promise and challenges of prompting with context information in noisy real-world settings. We publish code and data at this \href{https://github.com/tilmanbeck/stance-context-twitter}{page}.