Detecting Mental Manipulation in Speech via Synthetic Multi-Speaker Dialogue
This addresses the problem of identifying covert manipulation in speech for multimodal dialogue systems, but it is incremental as it extends existing text-based work to audio.
The study tackled the problem of detecting mental manipulation in spoken dialogues, a new task in computational social reasoning, by creating a synthetic multi-speaker benchmark and evaluating models and humans, finding that models had high specificity but lower recall on speech compared to text, and humans showed similar uncertainty.
Mental manipulation, the strategic use of language to covertly influence or exploit others, is a newly emerging task in computational social reasoning. Prior work has focused exclusively on textual conversations, overlooking how manipulative tactics manifest in speech. We present the first study of mental manipulation detection in spoken dialogues, introducing a synthetic multi-speaker benchmark SPEECHMENTALMANIP that augments a text-based dataset with high-quality, voice-consistent Text-to-Speech rendered audio. Using few-shot large audio-language models and human annotation, we evaluate how modality affects detection accuracy and perception. Our results reveal that models exhibit high specificity but markedly lower recall on speech compared to text, suggesting sensitivity to missing acoustic or prosodic cues in training. Human raters show similar uncertainty in the audio setting, underscoring the inherent ambiguity of manipulative speech. Together, these findings highlight the need for modality-aware evaluation and safety alignment in multimodal dialogue systems.