Adapting the Exploration Rate for Value-of-Information-Based Reinforcement Learning
This addresses the challenge of efficient exploration in reinforcement learning for game-playing agents, though it appears incremental as it builds on existing value-of-information methods.
The paper tackles the problem of adjusting the exploration rate in value-of-information-based reinforcement learning by converting it into a flow equilibrium problem and developing an efficient path-following scheme, resulting in better policies in fewer episodes than heuristic methods, with performance near or above human level in Nintendo GameBoy games.
In this paper, we consider the problem of adjusting the exploration rate when using value-of-information-based exploration. We do this by converting the value-of-information optimization into a problem of finding equilibria of a flow for a changing exploration rate. We then develop an efficient path-following scheme for converging to these equilibria and hence uncovering optimal action-selection policies. Under this scheme, the exploration rate is automatically adapted according to the agent's experiences. Global convergence is theoretically assured. We first evaluate our exploration-rate adaptation on the Nintendo GameBoy games Centipede and Millipede. We demonstrate aspects of the search process, like that it yields a hierarchy of state abstractions. We also show that our approach returns better policies in fewer episodes than conventional search strategies relying on heuristic, annealing-based exploration-rate adjustments. We then illustrate that these trends hold for deep, value-of-information-based agents that learn to play ten simple games and over forty more complicated games for the Nintendo GameBoy system. Performance either near or well above the level of human play is observed.