The Voice: Lessons on Trustworthy Conversational Agents from "Dune"
This addresses the problem of potential weaponization of AI for social manipulation by malicious actors, posing risks to public opinion and organizations, though it is incremental in exploring prevention strategies.
The paper tackles the threat of untrustworthy conversational agents for covert social manipulation, drawing inspiration from 'Dune' to highlight how generative AI enables scalable individualized influence, with models already capable of manipulating text, image, speech, and video.
The potential for untrustworthy conversational agents presents a significant threat for covert social manipulation. Taking inspiration from Frank Herbert's "Dune", where the Bene Gesserit Sisterhood uses the Voice for influence, manipulation, and control of people, we explore how generative AI provides a way to implement individualized influence at industrial scales. Already, these models can manipulate communication across text, image, speech, and most recently video. They are rapidly becoming affordable enough for any organization of even moderate means to train and deploy. If employed by malicious actors, they risk becoming powerful tools for shaping public opinion, sowing discord, and undermining organizations from companies to governments. As researchers and developers, it is crucial to recognize the potential for such weaponization and to explore strategies for prevention, detection, and defense against these emerging forms of sociotechnical manipulation.