Combining Transformer based Deep Reinforcement Learning with
  Black-Litterman Model for Portfolio Optimization
Ruoyu Sun Angelos Stefanidis Zhengyong Jiang Jionglong Su
As a model-free algorithm, deep reinforcement learning (DRL) agent learns and
makes decisions by interacting with the environment in an unsupervised way. In
recent years, DRL algorithms have been widely applied by scholars for portfolio
optimization in consecutive trading periods, since the DRL agent can
dynamically adapt to market changes and does not rely on the specification of
the joint dynamics across the assets. However, typical DRL agents for portfolio
optimization cannot learn a policy that is aware of the dynamic correlation
between portfolio asset returns. Since the dynamic correlations among portfolio
assets are crucial in optimizing the portfolio, the lack of such knowledge
makes it difficult for the DRL agent to maximize the return per unit of risk,
especially when the target market permits short selling (i.e., the US stock
market). In this research, we propose a hybrid portfolio optimization model
combining the DRL agent and the Black-Litterman (BL) model to enable the DRL
agent to learn the dynamic correlation between the portfolio asset returns and
implement an efficacious long/short strategy based on the correlation.
Essentially, the DRL agent is trained to learn the policy to apply the BL model
to determine the target portfolio weights. To test our DRL agent, we construct
the portfolio based on all the Dow Jones Industrial Average constitute stocks.
Empirical results of the experiments conducted on real-world United States
stock market data demonstrate that our DRL agent significantly outperforms
various comparison portfolio choice strategies and alternative DRL frameworks
by at least 42% in terms of accumulated return. In terms of the return per unit
of risk, our DRL agent significantly outperforms various comparative portfolio
choice strategies and alternative strategies based on other machine learning
frameworks.