Particle Swarm Optimization for Generating Interpretable Fuzzy Reinforcement Learning Policies
It addresses the problem of interpretable policy learning in safety-critical RL applications where online exploration is unsafe, though it appears incremental as it combines existing techniques like fuzzy controllers and model-based RL.
The paper tackles the challenge of automatically generating interpretable fuzzy controllers for reinforcement learning in safety-critical domains where online learning is prohibited, by introducing a model-based batch RL approach that trains on world models from transition samples, achieving high performance on standard benchmarks like mountain car and cart-pole tasks.
Fuzzy controllers are efficient and interpretable system controllers for continuous state and action spaces. To date, such controllers have been constructed manually or trained automatically either using expert-generated problem-specific cost functions or incorporating detailed knowledge about the optimal control strategy. Both requirements for automatic training processes are not found in most real-world reinforcement learning (RL) problems. In such applications, online learning is often prohibited for safety reasons because online learning requires exploration of the problem's dynamics during policy training. We introduce a fuzzy particle swarm reinforcement learning (FPSRL) approach that can construct fuzzy RL policies solely by training parameters on world models that simulate real system dynamics. These world models are created by employing an autonomous machine learning technique that uses previously generated transition samples of a real system. To the best of our knowledge, this approach is the first to relate self-organizing fuzzy controllers to model-based batch RL. Therefore, FPSRL is intended to solve problems in domains where online learning is prohibited, system dynamics are relatively easy to model from previously generated default policy transition samples, and it is expected that a relatively easily interpretable control policy exists. The efficiency of the proposed approach with problems from such domains is demonstrated using three standard RL benchmarks, i.e., mountain car, cart-pole balancing, and cart-pole swing-up. Our experimental results demonstrate high-performing, interpretable fuzzy policies.