CQM: Curriculum Reinforcement Learning with a Quantized World Model
This work addresses scalability issues in curriculum reinforcement learning for researchers and practitioners, though it is incremental as it builds on existing curriculum and world model techniques.
The paper tackles the challenge of generating curriculum goals in high-dimensional spaces for reinforcement learning by proposing a method that automatically defines a semantic goal space using vector quantized-variational autoencoders and graphs, and it outperforms state-of-the-art curriculum RL methods in data efficiency and performance on goal-reaching tasks.
Recent curriculum Reinforcement Learning (RL) has shown notable progress in solving complex tasks by proposing sequences of surrogate tasks. However, the previous approaches often face challenges when they generate curriculum goals in a high-dimensional space. Thus, they usually rely on manually specified goal spaces. To alleviate this limitation and improve the scalability of the curriculum, we propose a novel curriculum method that automatically defines the semantic goal space which contains vital information for the curriculum process, and suggests curriculum goals over it. To define the semantic goal space, our method discretizes continuous observations via vector quantized-variational autoencoders (VQ-VAE) and restores the temporal relations between the discretized observations by a graph. Concurrently, ours suggests uncertainty and temporal distance-aware curriculum goals that converges to the final goals over the automatically composed goal space. We demonstrate that the proposed method allows efficient explorations in an uninformed environment with raw goal examples only. Also, ours outperforms the state-of-the-art curriculum RL methods on data efficiency and performance, in various goal-reaching tasks even with ego-centric visual inputs.