Experimental Evidence that Empowerment May Drive Exploration in Sparse-Reward Environments
This work addresses exploration challenges in reinforcement learning for sparse-reward settings, but it is incremental as it builds on existing intrinsic motivation methods.
The study tackled the problem of reinforcement learning in sparse-reward environments by comparing an empowerment-inspired agent and a curious agent to a baseline, finding that both intrinsic reward agents performed similarly, providing experimental support for empowerment-driven exploration.
Reinforcement Learning (RL) is known to be often unsuccessful in environments with sparse extrinsic rewards. A possible countermeasure is to endow RL agents with an intrinsic reward function, or 'intrinsic motivation', which rewards the agent based on certain features of the current sensor state. An intrinsic reward function based on the principle of empowerment assigns rewards proportional to the amount of control the agent has over its own sensors. We implemented a variation on a recently proposed intrinsically motivated agent, which we refer to as the 'curious' agent, and an empowerment-inspired agent. The former leverages sensor state encoding with a variational autoencoder, while the latter predicts the next sensor state via a variational information bottleneck. We compared the performance of both agents to that of an advantage actor-critic baseline in four sparse reward grid worlds. Both the empowerment agent and its curious competitor seem to benefit to similar extents from their intrinsic rewards. This provides some experimental support to the conjecture that empowerment can be used to drive exploration.