Convex Q Learning in a Stochastic Environment: Extended Version
This provides a theoretical foundation for convex Q-learning in stochastic environments, potentially improving stability and convergence in reinforcement learning applications.
The paper introduces the first convex Q-learning formulation for Markov decision processes with function approximation, establishing conditions for bounded solutions and relationships with standard Q-learning, and demonstrates convergence with new rate analysis techniques.
The paper introduces the first formulation of convex Q-learning for Markov decision processes with function approximation. The algorithms and theory rest on a relaxation of a dual of Manne's celebrated linear programming characterization of optimal control. The main contributions firstly concern properties of the relaxation, described as a deterministic convex program: we identify conditions for a bounded solution, and a significant relationship between the solution to the new convex program, and the solution to standard Q-learning. The second set of contributions concern algorithm design and analysis: (i) A direct model-free method for approximating the convex program for Q-learning shares properties with its ideal. In particular, a bounded solution is ensured subject to a simple property of the basis functions; (ii) The proposed algorithms are convergent and new techniques are introduced to obtain the rate of convergence in a mean-square sense; (iii) The approach can be generalized to a range of performance criteria, and it is found that variance can be reduced by considering ``relative'' dynamic programming equations; (iv) The theory is illustrated with an application to a classical inventory control problem.