Requirements for Aligned, Dynamic Resolution of Conflicts in Operational Constraints
It addresses the challenge of ensuring AI systems make robust and aligned decisions in novel or under-specified contexts, which is incremental as it builds on existing work in autonomous systems and alignment.
This paper tackles the problem of autonomous AI systems encountering scenarios where no course of action fully satisfies all operational constraints, and it characterizes requirements for agents to construct, evaluate, and justify actions aligned with human expectations and values.
Deployed, autonomous AI systems must often evaluate multiple plausible courses of action (extended sequences of behavior) in novel or under-specified contexts. Despite extensive training, these systems will inevitably encounter scenarios where no available course of action fully satisfies all operational constraints (e.g., operating procedures, rules, laws, norms, and goals). To achieve goals in accordance with human expectations and values, agents must go beyond their trained policies and instead construct, evaluate, and justify candidate courses of action. These processes require contextual "knowledge" that may lie outside prior (policy) training. This paper characterizes requirements for agent decision making in these contexts. It also identifies the types of knowledge agents require to make decisions robust to agent goals and aligned with human expectations. Drawing on both analysis and empirical case studies, we examine how agents need to integrate normative, pragmatic, and situational understanding to select and then to pursue more aligned courses of action in complex, real-world environments.