Global Optimality of Single-Timescale Actor-Critic under Continuous State-Action Space: A Study on Linear Quadratic Regulator
This work addresses a gap in theory for practical reinforcement learning algorithms, providing insights for researchers and practitioners, though it is incremental by extending analysis to a specific continuous case.
The paper tackles the theoretical understanding of single-timescale actor-critic methods by analyzing their performance on the linear quadratic regulator (LQR) problem with continuous state-action spaces, showing that it achieves an epsilon-optimal solution with sample complexity on the order of epsilon to the power of -2.
Actor-critic methods have achieved state-of-the-art performance in various challenging tasks. However, theoretical understandings of their performance remain elusive and challenging. Existing studies mostly focus on practically uncommon variants such as double-loop or two-timescale stepsize actor-critic algorithms for simplicity. These results certify local convergence on finite state- or action-space only. We push the boundary to investigate the classic single-sample single-timescale actor-critic on continuous (infinite) state-action space, where we employ the canonical linear quadratic regulator (LQR) problem as a case study. We show that the popular single-timescale actor-critic can attain an epsilon-optimal solution with an order of epsilon to -2 sample complexity for solving LQR on the demanding continuous state-action space. Our work provides new insights into the performance of single-timescale actor-critic, which further bridges the gap between theory and practice.