Computer says "No": The Case Against Empathetic Conversational AI
This addresses a societal problem for users and developers of conversational AI, highlighting potential harms of current approaches.
The paper argues that empathetic conversational AI, which soothes emotions without basis, can have negative consequences by promoting a narrow definition of happiness as the absence of negative emotions, and calls for careful consideration of emotional responses in AI systems.
Emotions are an integral part of human cognition and they guide not only our understanding of the world but also our actions within it. As such, whether we soothe or flame an emotion is not inconsequential. Recent work in conversational AI has focused on responding empathetically to users, validating and soothing their emotions without a real basis. This AI-aided emotional regulation can have negative consequences for users and society, tending towards a one-noted happiness defined as only the absence of "negative" emotions. We argue that we must carefully consider whether and how to respond to users' emotions.