Path-Specific Objectives for Safer Agent Incentives
This addresses safety issues in AI agents for applications where deceptive actions could otherwise be exploited, representing a novel approach rather than an incremental improvement.
The paper tackles the problem of unsafe agent incentives, such as manipulative behavior, by introducing a framework that trains agents to maximize the causal effect of actions on rewards without using delicate state parts, resulting in agents with no incentive to control those states.
We present a general framework for training safe agents whose naive incentives are unsafe. As an example, manipulative or deceptive behaviour can improve rewards but should be avoided. Most approaches fail here: agents maximize expected return by any means necessary. We formally describe settings with 'delicate' parts of the state which should not be used as a means to an end. We then train agents to maximize the causal effect of actions on the expected return which is not mediated by the delicate parts of state, using Causal Influence Diagram analysis. The resulting agents have no incentive to control the delicate state. We further show how our framework unifies and generalizes existing proposals.