Semi-Gradient SARSA Routing with Theoretical Guarantee on Traffic Stability and Weight Convergence
This addresses traffic control problems in systems like transportation and data transmission, offering theoretical guarantees for stability and convergence, though it is incremental as it builds on reinforcement learning theory with new analysis methods.
The paper tackles dynamic routing over parallel servers by proposing a semi-gradient on-policy algorithm that learns an approximate optimal routing policy, proving traffic state stability and weight convergence theoretically and showing faster convergence than neural network-based methods with insignificant error in simulations.
We consider the traffic control problem of dynamic routing over parallel servers, which arises in a variety of engineering systems such as transportation and data transmission. We propose a semi-gradient, on-policy algorithm that learns an approximate optimal routing policy. The algorithm uses generic basis functions with flexible weights to approximate the value function across the unbounded state space. Consequently, the training process lacks Lipschitz continuity of the gradient, boundedness of the temporal-difference error, and a prior guarantee on ergodicity, which are the standard prerequisites in existing literature on reinforcement learning theory. To address this, we combine a Lyapunov approach and an ordinary differential equation-based method to jointly characterize the behavior of traffic state and approximation weights. Our theoretical analysis proves that the training scheme guarantees traffic state stability and ensures almost surely convergence of the weights to the approximate optimum. We also demonstrate via simulations that our algorithm attains significantly faster convergence than neural network-based methods with an insignificant approximation error.