Umbrella Reinforcement Learning -- computationally efficient tool for hard non-linear problems
This provides a more efficient solution for reinforcement learning practitioners dealing with challenging environments like sparse rewards and state traps.
The authors tackled hard nonlinear reinforcement learning problems with sparse rewards and state traps by combining umbrella sampling from physics/chemistry with optimal control methods, resulting in a computationally efficient approach that outperforms state-of-the-art algorithms in efficiency and universality.
We report a novel, computationally efficient approach for solving hard nonlinear problems of reinforcement learning (RL). Here we combine umbrella sampling, from computational physics/chemistry, with optimal control methods. The approach is realized on the basis of neural networks, with the use of policy gradient. It outperforms, by computational efficiency and implementation universality, all available state-of-the-art algorithms, in application to hard RL problems with sparse reward, state traps and lack of terminal states. The proposed approach uses an ensemble of simultaneously acting agents, with a modified reward which includes the ensemble entropy, yielding an optimal exploration-exploitation balance.