Hierarchical State Abstraction Based on Structural Information Principles
This addresses performance issues in reinforcement learning with rich observations, offering a general framework that can enhance various representation-learning objectives, though it appears incremental as it builds on existing state abstraction approaches.
The paper tackles the problem of essential information loss in state abstraction for reinforcement learning by proposing SISA, a framework based on structural information principles, which improves mean episode reward by up to 18.98% and sample efficiency by up to 44.44% compared to state-of-the-art methods.
State abstraction optimizes decision-making by ignoring irrelevant environmental information in reinforcement learning with rich observations. Nevertheless, recent approaches focus on adequate representational capacities resulting in essential information loss, affecting their performances on challenging tasks. In this article, we propose a novel mathematical Structural Information principles-based State Abstraction framework, namely SISA, from the information-theoretic perspective. Specifically, an unsupervised, adaptive hierarchical state clustering method without requiring manual assistance is presented, and meanwhile, an optimal encoding tree is generated. On each non-root tree node, a new aggregation function and condition structural entropy are designed to achieve hierarchical state abstraction and compensate for sampling-induced essential information loss in state abstraction. Empirical evaluations on a visual gridworld domain and six continuous control benchmarks demonstrate that, compared with five SOTA state abstraction approaches, SISA significantly improves mean episode reward and sample efficiency up to 18.98 and 44.44%, respectively. Besides, we experimentally show that SISA is a general framework that can be flexibly integrated with different representation-learning objectives to improve their performances further.