Principled Exploration via Optimistic Bootstrapping and Backward Induction
This addresses the problem of exploration inefficiency in deep reinforcement learning for AI agents, representing a novel method for a known bottleneck.
The paper tackled the challenge of efficient exploration in deep reinforcement learning by proposing Optimistic Bootstrapping and Backward Induction (OB2I), which constructs a UCB-bonus for optimistic exploration and improves sample-efficiency, outperforming state-of-the-art methods in MNIST maze and Atari experiments.
One principled approach for provably efficient exploration is incorporating the upper confidence bound (UCB) into the value function as a bonus. However, UCB is specified to deal with linear and tabular settings and is incompatible with Deep Reinforcement Learning (DRL). In this paper, we propose a principled exploration method for DRL through Optimistic Bootstrapping and Backward Induction (OB2I). OB2I constructs a general-purpose UCB-bonus through non-parametric bootstrap in DRL. The UCB-bonus estimates the epistemic uncertainty of state-action pairs for optimistic exploration. We build theoretical connections between the proposed UCB-bonus and the LSVI-UCB in a linear setting. We propagate future uncertainty in a time-consistent manner through episodic backward update, which exploits the theoretical advantage and empirically improves the sample-efficiency. Our experiments in the MNIST maze and Atari suite suggest that OB2I outperforms several state-of-the-art exploration approaches.