A Stochastic Composite Augmented Lagrangian Method For Reinforcement Learning
This addresses a computational bottleneck in reinforcement learning for large-scale environments, though it appears incremental as an enhancement to existing augmented Lagrangian methods.
The paper tackles the intractability of linear programming in deep reinforcement learning due to large state-action spaces and the double-sampling obstacle in augmented Lagrangian methods by proposing a deep parameterized approach that replaces conditional expectations with multipliers, showing theoretical convergence and competitive performance in experiments.
In this paper, we consider the linear programming (LP) formulation for deep reinforcement learning. The number of the constraints depends on the size of state and action spaces, which makes the problem intractable in large or continuous environments. The general augmented Lagrangian method suffers the double-sampling obstacle in solving the LP. Namely, the conditional expectations originated from the constraint functions and the quadratic penalties in the augmented Lagrangian function impose difficulties in sampling and evaluation. Motivated from the updates of the multipliers, we overcome the obstacles in minimizing the augmented Lagrangian function by replacing the intractable conditional expectations with the multipliers. Therefore, a deep parameterized augment Lagrangian method is proposed. Furthermore, the replacement provides a promising breakthrough to integrate the two steps in the augmented Lagrangian method into a single constrained problem. A general theoretical analysis shows that the solutions generated from a sequence of the constrained optimizations converge to the optimal solution of the LP if the error is controlled properly. A theoretical analysis on the quadratic penalty algorithm under neural tangent kernel setting shows the residual can be arbitrarily small if the parameter in network and optimization algorithm is chosen suitably. Preliminary experiments illustrate that our method is competitive to other state-of-the-art algorithms.