The Confusing Instance Principle for Online Linear Quadratic Control
This work addresses control problems in reinforcement learning, offering an incremental improvement over traditional methods like Optimism in the Face of Uncertainty and Thompson Sampling.
The paper tackles the problem of controlling linear systems with quadratic cost under unknown dynamics by proposing MED-LQ, a model-based reinforcement learning method based on the Confusing Instance principle, which achieves competitive performance in various scenarios.
We revisit the problem of controlling linear systems with quadratic cost under unknown dynamics with model-based reinforcement learning. Traditional methods like Optimism in the Face of Uncertainty and Thompson Sampling, rooted in multi-armed bandits (MABs), face practical limitations. In contrast, we propose an alternative based on the Confusing Instance (CI) principle, which underpins regret lower bounds in MABs and discrete Markov Decision Processes (MDPs) and is central to the Minimum Empirical Divergence (MED) family of algorithms, known for their asymptotic optimality in various settings. By leveraging the structure of LQR policies along with sensitivity and stability analysis, we develop MED-LQ. This novel control strategy extends the principles of CI and MED beyond small-scale settings. Our benchmarks on a comprehensive control suite demonstrate that MED-LQ achieves competitive performance in various scenarios while highlighting its potential for broader applications in large-scale MDPs.