Reinforcement Learning for Financial Index Tracking
Xianhua Peng Chenyin Gong Xue Dong He
We propose the first discrete-time infinite-horizon dynamic formulation of
the financial index tracking problem under both return-based tracking error and
value-based tracking error. The formulation overcomes the limitations of
existing models by incorporating the intertemporal dynamics of market
information variables not limited to prices, allowing exact calculation of
transaction costs, accounting for the tradeoff between overall tracking error
and transaction costs, allowing effective use of data in a long time period,
etc. The formulation also allows novel decision variables of cash injection or
withdraw. We propose to solve the portfolio rebalancing equation using a Banach
fixed point iteration, which allows to accurately calculate the transaction
costs specified as nonlinear functions of trading volumes in practice. We
propose an extension of deep reinforcement learning (RL) method to solve the
dynamic formulation. Our RL method resolves the issue of data limitation
resulting from the availability of a single sample path of financial data by a
novel training scheme. A comprehensive empirical study based on a 17-year-long
testing set demonstrates that the proposed method outperforms a benchmark
method in terms of tracking accuracy and has the potential for earning extra
profit through cash withdraw strategy.