Reinforcement-learning-based optimal trading in a simulated futures market with heterogeneous agents

Abstract

This paper simulates a futures market with multiple agents and sequential auctions, where agents receive long-lived heterogeneous signals on the true value of an asset and with a known deadline. The evolution of the amount of differential information and its impact on the distribution of overall gains and the pace of truth discovery is examined for various depth levels of the limit order book (LOB). The paper also formulates a dynamic programming model for the problem and presents an associated reinforcement learning (RL) algorithm for finding optimal strategy in exploiting informational disparity. This is done from the perspective of an agent whose information is superior to the collective information of the rest of the market. Finally, a numerical analysis is presented based on a futures market example to validate the proposed methodology for finding the optimal strategy. We find evidence in favor of a waiting strategy where agent does not reveal her signal until the last auction before the deadline. This result may help bring more insight into the micro-structural dynamics that work against market efficiency.