Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning
Provides foundational theory for state abstraction in RL, addressing a known bottleneck in scaling RL to complex systems.
The paper presents a unified compositional framework for specifying behavioral semantics in reinforcement learning, enabling safe transfer of behavioral structures (e.g., value functions, bisimulation) between abstract and concrete systems, with soundness guarantees for quantitative metrics.
State abstraction plays a key role in scaling reinforcement learning to complex but structured systems. In studying such systems, a wide range of behavioral structures have been studied in reinforcement learning, including value functions, invariants, bisimulation relations, and behavioral metrics. However, a general principle for determining what structures are provably preserved under state abstraction is still lacking. In this paper, we present a unified framework for defining and analyzing behavioral structures in reinforcement learning. Our framework provides a compositional way to specify behavioral semantics based on local, one-step descriptions of system dynamics. Using this framework, we establish results showing how behavioral structures can be safely transferred between abstract and concrete systems. We further show how to construct quantitative metrics from logical behavioral semantics with soundness guarantees. Together, these results provide a principled foundation for reasoning about behaviors under state abstraction in reinforcement learning and offer reusable definition and proof principles for a broad class of behavioral structures in reinforcement learning.