Learning Dynamic Abstract Representations for Sample-Efficient Reinforcement Learning
This addresses the challenge of sample inefficiency in RL for real-world applications by automating abstraction learning, though it is incremental as it builds on prior abstraction methods.
The paper tackles the problem of manually designing state abstractions in reinforcement learning by introducing a top-down approach that dynamically computes abstractions based on Q-value dispersion, resulting in significant sample efficiency gains and outperforming existing methods in multiple domains.
In many real-world problems, the learning agent needs to learn a problem's abstractions and solution simultaneously. However, most such abstractions need to be designed and refined by hand for different problems and domains of application. This paper presents a novel top-down approach for constructing state abstractions while carrying out reinforcement learning. Starting with state variables and a simulator, it presents a novel domain-independent approach for dynamically computing an abstraction based on the dispersion of Q-values in abstract states as the agent continues acting and learning. Extensive empirical evaluation on multiple domains and problems shows that this approach automatically learns abstractions that are finely-tuned to the problem, yield powerful sample efficiency, and result in the RL agent significantly outperforming existing approaches.