DPO: A Differential and Pointwise Control Approach to Reinforcement Learning
This addresses sample efficiency and physical consistency problems in scientific computing applications, representing a novel method for a known bottleneck rather than a foundational breakthrough.
The paper tackles the challenge of reinforcement learning in continuous state-action spaces for scientific computing by introducing Differential Reinforcement Learning, which reformulates RL from a continuous-time control perspective to embed physics priors and ensure consistent trajectories. The proposed Differential Policy Optimization algorithm achieves a theoretical regret bound of O(K^{5/6}) and empirically outperforms standard RL baselines on tasks like surface modeling and molecular dynamics under low-data conditions.
Reinforcement learning (RL) in continuous state-action spaces remains challenging in scientific computing due to poor sample efficiency and lack of pathwise physical consistency. We introduce Differential Reinforcement Learning (Differential RL), a novel framework that reformulates RL from a continuous-time control perspective via a differential dual formulation. This induces a Hamiltonian structure that embeds physics priors and ensures consistent trajectories without requiring explicit constraints. To implement Differential RL, we develop Differential Policy Optimization (DPO), a pointwise, stage-wise algorithm that refines local movement operators along the trajectory for improved sample efficiency and dynamic alignment. We establish pointwise convergence guarantees, a property not available in standard RL, and derive a competitive theoretical regret bound of $O(K^{5/6})$. Empirically, DPO outperforms standard RL baselines on representative scientific computing tasks, including surface modeling, grid control, and molecular dynamics, under low-data and physics-constrained conditions.