Who Does Your AI Work For? Designing Conversational Agents as Digital Fiduciaries
For users and designers of conversational AI, this work proposes a new normative baseline to ensure agents act in users' best interests, addressing a gap in current alignment-focused literature.
The paper argues that conversational AI agents, given their access to sensitive user data and anthropomorphic design, should be held to a fiduciary standard of care similar to professionals like lawyers or doctors, proposing fiduciary design as a guiding principle to unify trust and accountability.
Conversational agents are increasingly integrated into the most private and intimate aspects of users' lives, from discussions of mental health to financial decisions. As a result, these systems have access to reams of sensitive user data. Much of the literature on AI systems has focused on aligning users' goals with the agents that act on their behalf. While this work is vitally important, it may overlook the need to establish a new normative baseline. Conversational AI agents, designed to feel and interact anthropomorphically with human users, must be held to a standard of care commensurate with their capabilities and access. When a client hires a personal lawyer, undergoes surgery, or receives advice from an investment manager, the expert they consult often has a fiduciary duty to act in their client's best interests. This provocation argues that conversational agents should be held to a similar standard and introduces fiduciary design as a guiding principle. In this respect, conversational AI trust and accountability could be unified into a single design and legal paradigm.