Scalable and Safe Remediation of Defective Actions in Self-Learning Conversational Systems
This addresses the problem of ensuring safe and scalable policy updates in commercial conversational AI systems, though it appears incremental as it builds on existing off-policy evaluation and guard-railing techniques.
The paper tackles the challenge of balancing policy improvements and experience continuity in large-scale conversational AI systems by proposing a method that uses high-precision samples from historical regression incident reports to validate, safeguard, and improve policies before deployment, with the method being deployed in a production system to protect customers and enable long-term improvements.
Off-Policy reinforcement learning has been a driving force for the state-of-the-art conversational AIs leading to more natural humanagent interactions and improving the user satisfaction for goal-oriented agents. However, in large-scale commercial settings, it is often challenging to balance between policy improvements and experience continuity on the broad spectrum of applications handled by such system. In the literature, off-policy evaluation and guard-railing on aggregate statistics has been commonly used to address this problem. In this paper, we propose a method for curating and leveraging high-precision samples sourced from historical regression incident reports to validate, safe-guard, and improve policies prior to the online deployment. We conducted extensive experiments using data from a real-world conversational system and actual regression incidents. The proposed method is currently deployed in our production system to protect customers against broken experiences and enable long-term policy improvements.