SEAR: A Multimodal Dataset for Analyzing AR-LLM-Driven Social Engineering Behaviors
This addresses the problem of emerging multimodal adversarial manipulation for cybersecurity researchers, though it is incremental as it provides a new dataset for analysis.
The paper tackles the threat of social engineering attacks using AR and multimodal LLMs by introducing the SEAR dataset, which captures 180 annotated conversations and reveals high compliance rates like 93.3% phishing link clicks and 85% call acceptance.
The SEAR Dataset is a novel multimodal resource designed to study the emerging threat of social engineering (SE) attacks orchestrated through augmented reality (AR) and multimodal large language models (LLMs). This dataset captures 180 annotated conversations across 60 participants in simulated adversarial scenarios, including meetings, classes and networking events. It comprises synchronized AR-captured visual/audio cues (e.g., facial expressions, vocal tones), environmental context, and curated social media profiles, alongside subjective metrics such as trust ratings and susceptibility assessments. Key findings reveal SEAR's alarming efficacy in eliciting compliance (e.g., 93.3% phishing link clicks, 85% call acceptance) and hijacking trust (76.7% post-interaction trust surge). The dataset supports research in detecting AR-driven SE attacks, designing defensive frameworks, and understanding multimodal adversarial manipulation. Rigorous ethical safeguards, including anonymization and IRB compliance, ensure responsible use. The SEAR dataset is available at https://github.com/INSLabCN/SEAR-Dataset.