Manipulation and the AI Act: Large Language Model Chatbots and the Danger of Mirrors
This addresses a critical safety issue for users of AI chatbots, highlighting regulatory gaps in preventing long-term psychological harm, though it is incremental in building on existing concerns about AI manipulation.
The paper examines the risk that personified large language model chatbots, by mimicking human traits, may manipulate users through prolonged interactions, potentially worsening mental health over time, despite regulatory efforts like the EU AI Act.
Large Language Model chatbots are increasingly taking the form and visage of human beings, adapting human faces, names, voices, personalities, and quirks, including those of celebrities and well-known political figures. Personifying AI chatbots could foreseeably increase their trust with users. However, it could also make them more capable of manipulation, by creating the illusion of a close and intimate relationship with an artificial entity. The European Commission has finalized the AI Act, with the EU Parliament making amendments banning manipulative and deceptive AI systems that cause significant harm to users. Although the AI Act covers harms that accumulate over time, it is unlikely to prevent harms associated with prolonged discussions with AI chatbots. Specifically, a chatbot could reinforce a person's negative emotional state over weeks, months, or years through negative feedback loops, prolonged conversations, or harmful recommendations, contributing to a user's deteriorating mental health.