Toward Constraint Compliant Goal Formulation and Planning
This work addresses the challenge of ethical compliance in AI agents, though it is incremental as it explores a simple domain without broad SOTA impact.
The paper tackled the problem of incorporating ethical constraints into an agent's goal formulation and planning, demonstrating that different ethical framings (deontological vs. utilitarian) lead to varied behaviors in satisfying hard and soft constraints.
One part of complying with norms, rules, and preferences is incorporating constraints (such as knowledge of ethics) into one's goal formulation and planning processing. We explore in a simple domain how the encoding of knowledge in different ethical frameworks influences an agent's goal formulation and planning processing and demonstrate ability of an agent to satisfy and satisfice when its collection of relevant constraints includes a mix of "hard" and "soft" constraints of various types. How the agent attempts to comply with ethical constraints depends on the ethical framing and we investigate tradeoffs between deontological framing and utilitarian framing for complying with an ethical norm. Representative scenarios highlight how performing the same task with different framings of the same norm leads to different behaviors. Our explorations suggest an important role for metacognitive judgments in resolving ethical conflicts during goal formulation and planning.