Reinforcement Learning with $ω$-Regular Objectives and Constraints
This addresses the problem of safety-performance trade-offs in RL for applications requiring precise temporal and safety-critical specifications, though it is incremental as it builds on existing ω-regular frameworks.
The paper tackles the limitations of scalar rewards in reinforcement learning by combining ω-regular objectives with explicit constraints to separate safety requirements and optimization targets, resulting in a model-based RL algorithm that maximizes satisfaction probability while adhering to constraints within thresholds.
Reinforcement learning (RL) commonly relies on scalar rewards with limited ability to express temporal, conditional, or safety-critical goals, and can lead to reward hacking. Temporal logic expressible via the more general class of $ω$-regular objectives addresses this by precisely specifying rich behavioural properties. Even still, measuring performance by a single scalar (be it reward or satisfaction probability) masks safety-performance trade-offs that arise in settings with a tolerable level of risk. We address both limitations simultaneously by combining $ω$-regular objectives with explicit constraints, allowing safety requirements and optimisation targets to be treated separately. We develop a model-based RL algorithm based on linear programming, which in the limit produces a policy maximising the probability of satisfying an $ω$-regular objective while also adhering to $ω$-regular constraints within specified thresholds. Furthermore, we establish a translation to constrained limit-average problems with optimality-preserving guarantees.