Towards Generalizable Reinforcement Learning for Trade Execution
Chuheng Zhang Yitong Duan Xiaoyu Chen Jianyu Chen Jian Li Li Zhao
Optimized trade execution is to sell (or buy) a given amount of assets in a
given time with the lowest possible trading cost. Recently, reinforcement
learning (RL) has been applied to optimized trade execution to learn smarter
policies from market data. However, we find that many existing RL methods
exhibit considerable overfitting which prevents them from real deployment. In
this paper, we provide an extensive study on the overfitting problem in
optimized trade execution. First, we model the optimized trade execution as
offline RL with dynamic context (ORDC), where the context represents market
variables that cannot be influenced by the trading policy and are collected in
an offline manner. Under this framework, we derive the generalization bound and
find that the overfitting issue is caused by large context space and limited
context samples in the offline setting. Accordingly, we propose to learn
compact representations for context to address the overfitting problem, either
by leveraging prior knowledge or in an end-to-end manner. To evaluate our
algorithms, we also implement a carefully designed simulator based on
historical limit order book (LOB) data to provide a high-fidelity benchmark for
different algorithms. Our experiments on the high-fidelity simulator
demonstrate that our algorithms can effectively alleviate overfitting and
achieve better performance.