Finite Time Analysis of Constrained Natural Critic-Actor Algorithm with Improved Sample Complexity
This work addresses constrained reinforcement learning for safety-critical applications, offering incremental improvements in sample complexity and convergence analysis.
The paper tackles the problem of developing a natural critic-actor algorithm with function approximation for constrained long-run average cost settings, providing non-asymptotic convergence guarantees with optimal learning rates and improved sample complexity, and shows competitive performance in Safety-Gym experiments.
Recent studies have increasingly focused on non-asymptotic convergence analyses for actor-critic (AC) algorithms. One such effort introduced a two-timescale critic-actor algorithm for the discounted cost setting using a tabular representation, where the usual roles of the actor and critic are reversed. However, only asymptotic convergence was established there. Subsequently, both asymptotic and non-asymptotic analyses of the critic-actor algorithm with linear function approximation were conducted. In our work, we introduce the first natural critic-actor algorithm with function approximation for the long-run average cost setting and under inequality constraints. We provide the non-asymptotic convergence guarantees for this algorithm. Our analysis establishes optimal learning rates and we also propose a modification to enhance sample complexity. We further show the results of experiments on three different Safety-Gym environments where our algorithm is found to be competitive in comparison with other well known algorithms.