Expectation Alignment: Handling Reward Misspecification in the Presence of Expectation Mismatch
This work addresses a central challenge in AI safety by providing a formal framework for handling reward misspecification, which is incremental as it builds on existing theories to offer new insights and solutions.
The paper tackles the problem of reward misspecification in AI safety by introducing the Expectation Alignment framework, which uses theory of mind to formally define and address mismatches between user expectations and AI objectives, and proposes an interactive algorithm that infers user expectations from specified rewards, evaluated on MDP benchmarks.
Detecting and handling misspecified objectives, such as reward functions, has been widely recognized as one of the central challenges within the domain of Artificial Intelligence (AI) safety research. However, even with the recognition of the importance of this problem, we are unaware of any works that attempt to provide a clear definition for what constitutes (a) misspecified objectives and (b) successfully resolving such misspecifications. In this work, we use the theory of mind, i.e., the human user's beliefs about the AI agent, as a basis to develop a formal explanatory framework called Expectation Alignment (EAL) to understand the objective misspecification and its causes. Our EAL framework not only acts as an explanatory framework for existing works but also provides us with concrete insights into the limitations of existing methods to handle reward misspecification and novel solution strategies. We use these insights to propose a new interactive algorithm that uses the specified reward to infer potential user expectations about the system behavior. We show how one can efficiently implement this algorithm by mapping the inference problem into linear programs. We evaluate our method on a set of standard Markov Decision Process (MDP) benchmarks.