Can Reinforcement Learning Efficiently Discover Price Manipulation?
For financial regulators and market participants, this work highlights the potential of RL to uncover manipulation strategies, but also warns of risks when deploying learning algorithms without safeguards.
This paper investigates whether a model-free reinforcement learning agent can discover price manipulation strategies more effectively than a model-based approach with noisy parameter estimates. In an Almgren-Chriss framework, RL outperforms the model-based method for intermediate volatility, even with limited data, but fails for large volatility and is outperformed for small volatility.
In this paper, we investigate whether a model-free RL agent can identify and exploit price manipulation opportunities more effectively than a traditional model-based approach that assumes correct specification of the data-generating process but relies on noisy parameter estimates. We consider a single-asset market in which prices evolve according to an Almgren-Chriss framework with non-linear permanent impact and linear temporary impact. We first establish the existence of price-manipulative strategies in discrete time and compute the optimal benchmark strategy using Sequential Least Squares Quadratic Programming under full information. We then compare two finite-sample learning approaches: a model-based procedure that estimates impact parameters from simulated execution data and an agnostic RL approach based on Deep Deterministic Policy Gradient, trained directly on the same amount of data. For intermediate volatility, the RL agent successfully discovers profitable manipulative strategies without explicit knowledge of the underlying model, even when training data are quite limited. More importantly, RL consistently outperforms the model-based approach when parameter estimates are affected by sampling error, despite the latter benefiting from the correct model specification. For large volatility, all methods are unable to identify manipulation opportunities, while for small volatility, the model based approach outperforms RL. These findings highlight both the effectiveness of RL in complex control problems and the risks associated with deploying learning algorithms in financial markets without appropriate safeguards.