Stratifying Reinforcement Learning with Signal Temporal Logic
This provides a theoretical framework for understanding the geometry of decision spaces in reinforcement learning, though it appears incremental as it builds on existing STL and stratification concepts.
The paper tackles the problem of analyzing deep reinforcement learning embedding spaces by developing a stratification-based semantics for Signal Temporal Logic, revealing a correspondence between stratification theory and STL formulas and applying it to Minigrid games with preliminary evidence of computationally efficient signatures.
In this paper, we develop a stratification-based semantics for Signal Temporal Logic (STL) in which each atomic predicate is interpreted as a membership test in a stratified space. This perspective reveals a novel correspondence principle between stratification theory and STL, showing that most STL formulas can be viewed as inducing a stratification of space-time. The significance of this interpretation is twofold. First, it offers a fresh theoretical framework for analyzing the structure of the embedding space generated by deep reinforcement learning (DRL) and relates it to the geometry of the ambient decision space. Second, it provides a principled framework that both enables the reuse of existing high-dimensional analysis tools and motivates the creation of novel computational techniques. To ground the theory, we (1) illustrate the role of stratification theory in Minigrid games and (2) apply numerical techniques to the latent embeddings of a DRL agent playing such a game where the robustness of STL formulas is used as the reward. In the process, we propose computationally efficient signatures that, based on preliminary evidence, appear promising for uncovering the stratification structure of such embedding spaces.