Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling
This work tackles the critical problem of mitigating potential harms from conversational AI for researchers and developers, though it is incremental as it builds on existing safety and design principles.
The paper addresses the safety challenges of releasing end-to-end conversational AI models that may learn harmful behaviors from training data, proposing a framework based on value-sensitive design and a suite of tools to guide researchers in making informed release decisions.
Over the last several years, end-to-end neural conversational agents have vastly improved in their ability to carry a chit-chat conversation with humans. However, these models are often trained on large datasets from the internet, and as a result, may learn undesirable behaviors from this data, such as toxic or otherwise harmful language. Researchers must thus wrestle with the issue of how and when to release these models. In this paper, we survey the problem landscape for safety for end-to-end conversational AI and discuss recent and related work. We highlight tensions between values, potential positive impact and potential harms, and provide a framework for making decisions about whether and how to release these models, following the tenets of value-sensitive design. We additionally provide a suite of tools to enable researchers to make better-informed decisions about training and releasing end-to-end conversational AI models.