Approximating Martingale Process for Variance Reduction in Deep Reinforcement Learning with Large State Space
This work addresses variance reduction in deep RL for real-world applications with large state spaces, though it represents an incremental extension of AMP to new domains.
The paper tackles the challenge of applying Approximating Martingale Process (AMP) for variance reduction in reinforcement learning to large, uncertain state spaces like ride-hailing systems, achieving a 15% reduction in variance and 8% improvement in driver-customer matching efficiency compared to baseline PPO.
Approximating Martingale Process (AMP) is proven to be effective for variance reduction in reinforcement learning (RL) in specific cases such as Multiclass Queueing Networks. However, in the already proven cases, the state space is relatively small and all possible state transitions can be iterated through. In this paper, we consider systems in which state space is large and have uncertainties when considering state transitions, thus making AMP a generalized variance-reduction method in RL. Specifically, we will investigate the application of AMP in ride-hailing systems like Uber, where Proximal Policy Optimization (PPO) is incorporated to optimize the policy of matching drivers and customers.