DeepSeekMath Meets Order Book: Group-Aware Policy Optimization for High-Frequency Directional Trading
For practitioners in high-frequency trading, this work demonstrates that group-normalized policy optimization outperforms value-based methods on order-flow state representations, though results are limited to a simplified backtesting setup.
This paper applies policy-gradient reinforcement learning to high-frequency trading on limit order books, using group-aware PPO variants (GRPO, GSPO) that improve net average PnL, profitability, and drawdown over Q-learning baselines on AMZN, AAPL, and GOOG backtests.
This paper studies reinforcement learning for high-frequency trading on limit order books by pairing an Order-Flow-based state model with policy-gradient methods. Instead of value-based RL techniques like tabular Q-learning, our approach deploys policy-based methods like vanilla PPO and DeepSeekMath-inspired variants like GRPO and GSPO, that use group-normalized updates and downside-aware shaping. On backtests with financial assets AMZN, AAPL, and GOOG under a simplified backtesting setup based on spread-scaled rewards, these new policies improve net average PnL, profitability, and drawdown over the Q-Learning baseline. Our results show that (1) Order-Flow signals are an adequate state for policy RL and (2) group-aware PPO surrogates are preferable over value-based baselines.