Dual Control for Approximate Bayesian Reinforcement Learning
This work addresses the exploration-exploitation trade-off in reinforcement learning for researchers, offering an incremental extension of existing methods to approximate Bayesian RL.
The paper tackles the problem of controlling non-episodic, finite-horizon dynamical systems with uncertain dynamics by extending an approximate dual control approach from control theory to modern regression methods like generalized linear regression. Experiments on simulated systems show that this framework provides a useful approximation to intractable Bayesian reinforcement learning, producing structured exploration strategies that differ from standard RL approaches.
Control of non-episodic, finite-horizon dynamical systems with uncertain dynamics poses a tough and elementary case of the exploration-exploitation trade-off. Bayesian reinforcement learning, reasoning about the effect of actions and future observations, offers a principled solution, but is intractable. We review, then extend an old approximate approach from control theory---where the problem is known as dual control---in the context of modern regression methods, specifically generalized linear regression. Experiments on simulated systems show that this framework offers a useful approximation to the intractable aspects of Bayesian RL, producing structured exploration strategies that differ from standard RL approaches. We provide simple examples for the use of this framework in (approximate) Gaussian process regression and feedforward neural networks for the control of exploration.