Finite Horizon Q-learning: Stability, Convergence, Simulations and an application on Smart Grids
This work addresses the lack of analysis for Q-learning in finite horizon settings, which is important for applications like smart grids, though it is incremental as it adapts an existing method to a new context.
The authors developed a Q-learning algorithm for finite horizon Markov decision processes and provided a full proof of its stability and convergence using the O.D.E. method, demonstrating performance on random MDPs and a smart grid application.
Q-learning is a popular reinforcement learning algorithm. This algorithm has however been studied and analysed mainly in the infinite horizon setting. There are several important applications which can be modeled in the framework of finite horizon Markov decision processes. We develop a version of Q-learning algorithm for finite horizon Markov decision processes (MDP) and provide a full proof of its stability and convergence. Our analysis of stability and convergence of finite horizon Q-learning is based entirely on the ordinary differential equations (O.D.E) method. We also demonstrate the performance of our algorithm on a setting of random MDP as well as on an application on smart grids.