Foundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf restraining specifications
This addresses the challenge of ensuring AI safety by integrating external specifications, though it appears incremental as it builds on existing logical frameworks and reinforcement learning methods.
The paper tackles the problem of aligning reinforcement learning agents with logical constraints (LTLf/LDLf) from an external authority, showing that agents can learn to conform to these constraints under general conditions.
In this work we investigate on the concept of "restraining bolt", envisioned in Science Fiction. Specifically we introduce a novel problem in AI. We have two distinct sets of features extracted from the world, one by the agent and one by the authority imposing restraining specifications (the "restraining bolt"). The two sets are apparently unrelated since of interest to independent parties, however they both account for (aspects of) the same world. We consider the case in which the agent is a reinforcement learning agent on the first set of features, while the restraining bolt is specified logically using linear time logic on finite traces LTLf/LDLf over the second set of features. We show formally, and illustrate with examples, that, under general circumstances, the agent can learn while shaping its goals to suitably conform (as much as possible) to the restraining bolt specifications.