Anticipatory Reinforcement Learning: From Generative Path-Laws to Distributional Value Functions
This addresses the challenge of accurate foresight and stability in volatile, continuous-time environments for reinforcement learning applications, representing a novel method rather than an incremental improvement.
The paper tackles the problem of reinforcement learning in non-Markovian environments with jump-diffusions and structural breaks, where traditional methods fail, by introducing Anticipatory Reinforcement Learning (ARL) to bridge this gap using a signature-augmented manifold and self-consistent field approach, resulting in reduced computational complexity and variance while enabling proactive risk management and superior policy stability.
This paper introduces Anticipatory Reinforcement Learning (ARL), a novel framework designed to bridge the gap between non-Markovian decision processes and classical reinforcement learning architectures, specifically under the constraint of a single observed trajectory. In environments characterised by jump-diffusions and structural breaks, traditional state-based methods often fail to capture the essential path-dependent geometry required for accurate foresight. We resolve this by lifting the state space into a signature-augmented manifold, where the history of the process is embedded as a dynamical coordinate. By utilising a self-consistent field approach, the agent maintains an anticipated proxy of the future path-law, allowing for a deterministic evaluation of expected returns. This transition from stochastic branching to a single-pass linear evaluation significantly reduces computational complexity and variance. We prove that this framework preserves fundamental contraction properties and ensures stable generalisation even in the presence of heavy-tailed noise. Our results demonstrate that by grounding reinforcement learning in the topological features of path-space, agents can achieve proactive risk management and superior policy stability in highly volatile, continuous-time environments.