This paper investigates whether a model-free Reinforcement Learning (RL) agent can more effectively identify and exploit price manipulation opportunities than a traditional model-based approach, which assumes correct specification of the data-generating process but relies on noisy parameter estimates. We consider a single-asset market where prices evolve according to the Almgren-Chriss framework, featuring non-linear permanent impact and linear temporary impact.
First, we establish the existence of price-manipulative strategies in discrete time and compute the optimal benchmark strategy using Sequential Least Squares Quadratic Programming under full information. We then compare two finite-sample learning approaches: a model-based procedure that estimates impact parameters from simulated execution data and an agnostic RL approach based on Deep Deterministic Policy Gradient, trained directly on the same amount of data.
For intermediate volatility, the RL agent successfully discovers profitable manipulative strategies without explicit knowledge of the underlying model, even when training data is quite limited. More importantly, RL consistently outperforms the model-based approach when parameter estimates are affected by sampling error, despite the latter benefiting from correct model specification. For large volatility, all methods fail to identify manipulation opportunities, while for small volatility, the model-based approach outperforms RL. These findings highlight both the effectiveness of RL in complex control problems and the risks associated with deploying learning algorithms in financial markets without appropriate safeguards.
Blogger's Review: This study showcases the potential of reinforcement learning in identifying manipulation in financial markets, especially under data scarcity. It provides a significant theoretical foundation for future financial algorithm design but also reminds us to exercise caution when applying RL to avoid potential market risks.