Fractional Policy Gradients: Reinforcement Learning with Long-Term Memory
This work addresses a foundational issue in reinforcement learning for agents requiring long-term memory, offering a mathematically grounded solution with significant performance improvements.
The paper tackles the problem of high variance and inefficient sampling in reinforcement learning due to Markovian assumptions by introducing Fractional Policy Gradients (FPG), which uses fractional calculus for long-term temporal modeling, resulting in 35-68% sample efficiency gains and 24-52% variance reduction compared to state-of-the-art baselines.
We propose Fractional Policy Gradients (FPG), a reinforcement learning framework incorporating fractional calculus for long-term temporal modeling in policy optimization. Standard policy gradient approaches face limitations from Markovian assumptions, exhibiting high variance and inefficient sampling. By reformulating gradients using Caputo fractional derivatives, FPG establishes power-law temporal correlations between state transitions. We develop an efficient recursive computation technique for fractional temporal-difference errors with constant time and memory requirements. Theoretical analysis shows FPG achieves asymptotic variance reduction of order O(t^(-alpha)) versus standard policy gradients while preserving convergence. Empirical validation demonstrates 35-68% sample efficiency gains and 24-52% variance reduction versus state-of-the-art baselines. This framework provides a mathematically grounded approach for leveraging long-range dependencies without computational overhead.