Intrinsic Exploration as Multi-Objective RL
This addresses the challenge of sparse rewards in RL for domains like robotics, offering a more controlled and scalable exploration approach, though it is incremental in improving existing intrinsic methods.
The paper tackles the problem of sub-optimal exploration in reinforcement learning by proposing a multi-objective RL framework that optimizes exploration and exploitation as separate objectives, resulting in EMU-Q outperforming classic and intrinsic methods on continuous control benchmarks and a robotic manipulator.
Intrinsic motivation enables reinforcement learning (RL) agents to explore when rewards are very sparse, where traditional exploration heuristics such as Boltzmann or e-greedy would typically fail. However, intrinsic exploration is generally handled in an ad-hoc manner, where exploration is not treated as a core objective of the learning process; this weak formulation leads to sub-optimal exploration performance. To overcome this problem, we propose a framework based on multi-objective RL where both exploration and exploitation are being optimized as separate objectives. This formulation brings the balance between exploration and exploitation at a policy level, resulting in advantages over traditional methods. This also allows for controlling exploration while learning, at no extra cost. Such strategies achieve a degree of control over agent exploration that was previously unattainable with classic or intrinsic rewards. We demonstrate scalability to continuous state-action spaces by presenting a method (EMU-Q) based on our framework, guiding exploration towards regions of higher value-function uncertainty. EMU-Q is experimentally shown to outperform classic exploration techniques and other intrinsic RL methods on a continuous control benchmark and on a robotic manipulator.