LTL-Constrained Steady-State Policy Synthesis
This work addresses the challenge of integrating multiple specification types for policy synthesis in AI and control systems, offering a unified solution that is incremental in combining existing methods.
The paper tackles the problem of synthesizing decision-making policies for agents under combined qualitative (LTL), quantitative (steady-state), and reward specifications in Markov decision processes, resulting in an algorithm that maximizes reward while ensuring LTL probability and steady-state constraints, with runtime polynomial in MDP and automaton sizes.
Decision-making policies for agents are often synthesized with the constraint that a formal specification of behaviour is satisfied. Here we focus on infinite-horizon properties. On the one hand, Linear Temporal Logic (LTL) is a popular example of a formalism for qualitative specifications. On the other hand, Steady-State Policy Synthesis (SSPS) has recently received considerable attention as it provides a more quantitative and more behavioural perspective on specifications, in terms of the frequency with which states are visited. Finally, rewards provide a classic framework for quantitative properties. In this paper, we study Markov decision processes (MDP) with the specification combining all these three types. The derived policy maximizes the reward among all policies ensuring the LTL specification with the given probability and adhering to the steady-state constraints. To this end, we provide a unified solution reducing the multi-type specification to a multi-dimensional long-run average reward. This is enabled by Limit-Deterministic Büchi Automata (LDBA), recently studied in the context of LTL model checking on MDP, and allows for an elegant solution through a simple linear programme. The algorithm also extends to the general $ω$-regular properties and runs in time polynomial in the sizes of the MDP as well as the LDBA.