Augmented RBMLE-UCB Approach for Adaptive Control of Linear Quadratic Systems
This work addresses the adaptive LQ control problem for stochastic systems, offering a novel algorithm that combines historical and modern optimism-based approaches, with incremental improvements in empirical performance over existing methods.
The paper tackles the adaptive control of unknown stochastic linear systems with quadratic costs by reconciling the Reward Biased Maximum Likelihood Estimate (RBMLE) and Upper Confidence Bound (UCB) methods, proposing an Augmented RBMLE-UCB algorithm that achieves the best-known theoretical regret of $\Tilde{\mathcal{O}}(\sqrt{T})$ and empirically outperforms several benchmarks, such as UCB and Thompson Sampling, by a huge margin in simulations including flight control examples.
We consider the problem of controlling an unknown stochastic linear system with quadratic costs - called the adaptive LQ control problem. We re-examine an approach called ''Reward Biased Maximum Likelihood Estimate'' (RBMLE) that was proposed more than forty years ago, and which predates the ''Upper Confidence Bound'' (UCB) method as well as the definition of ''regret'' for bandit problems. It simply added a term favoring parameters with larger rewards to the criterion for parameter estimation. We show how the RBMLE and UCB methods can be reconciled, and thereby propose an Augmented RBMLE-UCB algorithm that combines the penalty of the RBMLE method with the constraints of the UCB method, uniting the two approaches to optimism in the face of uncertainty. We establish that theoretically, this method retains $\Tilde{\mathcal{O}}(\sqrt{T})$ regret, the best-known so far. We further compare the empirical performance of the proposed Augmented RBMLE-UCB and the standard RBMLE (without the augmentation) with UCB, Thompson Sampling, Input Perturbation, Randomized Certainty Equivalence and StabL on many real-world examples including flight control of Boeing 747 and Unmanned Aerial Vehicle. We perform extensive simulation studies showing that the Augmented RBMLE consistently outperforms UCB, Thompson Sampling and StabL by a huge margin, while it is marginally better than Input Perturbation and moderately better than Randomized Certainty Equivalence.