A relaxed technical assumption for posterior sampling-based reinforcement learning for control of unknown linear systems
This is an incremental improvement for researchers in reinforcement learning and control theory, as it simplifies the theoretical conditions for an existing algorithm without changing its performance.
The paper tackles the problem of controlling unknown linear quadratic systems by relaxing a technical assumption in a Thompson sampling algorithm, showing that a minor modification allows replacing a norm-based assumption with a milder spectral radius condition while maintaining the same Bayesian regret bound of $ ilde{\mathcal{O}}(\sqrt{T})$.
We revisit the Thompson sampling algorithm to control an unknown linear quadratic (LQ) system recently proposed by Ouyang et al (arXiv:1709.04047). The regret bound of the algorithm was derived under a technical assumption on the induced norm of the closed loop system. In this technical note, we show that by making a minor modification in the algorithm (in particular, ensuring that an episode does not end too soon), this technical assumption on the induced norm can be replaced by a milder assumption in terms of the spectral radius of the closed loop system. The modified algorithm has the same Bayesian regret of $\tilde{\mathcal{O}}(\sqrt{T})$, where $T$ is the time-horizon and the $\tilde{\mathcal{O}}(\cdot)$ notation hides logarithmic terms in~$T$.