Unraveling SITT: Social Influence Technique Taxonomy and Detection with LLMs
This work addresses the challenge of identifying nuanced social influence in dialogues for fields like AI safety and social science, though it is incremental in benchmarking existing models.
The researchers tackled the problem of detecting subtle social influence techniques in text by creating the SITT taxonomy and dataset, and found that LLMs like Claude 3.5 achieved moderate success with an F1 score of 0.45 for categories, but overall performance was limited.
In this work we present the Social Influence Technique Taxonomy (SITT), a comprehensive framework of 58 empirically grounded techniques organized into nine categories, designed to detect subtle forms of social influence in textual content. We also investigate the LLMs ability to identify various forms of social influence. Building on interdisciplinary foundations, we construct the SITT dataset -- a 746-dialogue corpus annotated by 11 experts in Polish and translated into English -- to evaluate the ability of LLMs to identify these techniques. Using a hierarchical multi-label classification setup, we benchmark five LLMs, including GPT-4o, Claude 3.5, Llama-3.1, Mixtral, and PLLuM. Our results show that while some models, notably Claude 3.5, achieved moderate success (F1 score = 0.45 for categories), overall performance of models remains limited, particularly for context-sensitive techniques. The findings demonstrate key limitations in current LLMs' sensitivity to nuanced linguistic cues and underscore the importance of domain-specific fine-tuning. This work contributes a novel resource and evaluation example for understanding how LLMs detect, classify, and potentially replicate strategies of social influence in natural dialogues.