Categorical semantics of compositional reinforcement learning
This work addresses the need for modular, interpretable, and safe task specifications in reinforcement learning, though it appears incremental by building on existing categorical and abstraction theories.
The paper tackles the problem of generating compositional knowledge representations in reinforcement learning by developing a categorical semantics framework based on the category MDP, which models tasks as Markov decision processes and uses pushout operations for compositionality, resulting in a unified theory that generalizes previous abstraction methods.
Compositional knowledge representations in reinforcement learning (RL) facilitate modular, interpretable, and safe task specifications. However, generating compositional models requires the characterization of minimal assumptions for the robustness of the compositionality feature, especially in the case of functional decompositions. Using a categorical point of view, we develop a knowledge representation framework for a compositional theory of RL. Our approach relies on the theoretical study of the category MDP, whose objects are Markov decision processes (MDPs) acting as models of tasks. The categorical semantics models the compositionality of tasks through the application of pushout operations akin to combining puzzle pieces. As a practical application of these pushout operations, we introduce zig-zag diagrams that rely on the compositional guarantees engendered by the category MDP. We further prove that properties of the category MDP unify concepts, such as enforcing safety requirements and exploiting symmetries, generalizing previous abstraction theories for RL.