Muesli: Combining Improvements in Policy Optimization
This work addresses the challenge of efficient and high-performing reinforcement learning for applications like gaming and control, though it appears incremental as it builds on existing methods.
The authors tackled the problem of improving policy optimization in reinforcement learning by combining regularized policy optimization with model learning as an auxiliary loss, achieving state-of-the-art performance on Atari without deep search.
We propose a novel policy update that combines regularized policy optimization with model learning as an auxiliary loss. The update (henceforth Muesli) matches MuZero's state-of-the-art performance on Atari. Notably, Muesli does so without using deep search: it acts directly with a policy network and has computation speed comparable to model-free baselines. The Atari results are complemented by extensive ablations, and by additional results on continuous control and 9x9 Go.