Model-Free Reinforcement Learning for Financial Portfolios: A Brief Survey
---
paper_title: Theory of rational option pricing

paper_content:

AbstractThe following sections are included:IntroductionRestrictions on rational option pricingEffects of dividends and changing exercise priceRestrictions on rational put option oricinaRational option pricing along Black-Scholes linesAn alternative derivation of the Black-Scholes modelExtension of the model to include dividend payments and exercise price changesValuing an American put optionValuing the “down and-out” call optionValuing a callable warrantAppendix 1Appendix 2Referencesdiscussion: Option Pricing Theory and Its ApplicationsINTRODUCTIONTHE MARTINGALE APPROACH TO OPTION PRICINGThe SetupDynamic Spanning and the Martingale Representation TheoremSome GeneralizationsEXISTENCE AND PROPERTIES OF OPTIMAL STRATEGIESAPPLICATIONS TO CONTINGENT-CLAIM PRICINGNOTESREFERENCES
---
paper_title: Human-level control through deep reinforcement learning

paper_content:

The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.
---
paper_title: Proximal Policy Optimization Algorithms

paper_content:

We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.
---
paper_title: Mastering the game of Go with deep neural networks and tree search

paper_content:

The game of Go has long been viewed as the most challenging of classic games for artificial intelligence owing to its enormous search space and the difficulty of evaluating board positions and moves. Here we introduce a new approach to computer Go that uses ‘value networks’ to evaluate board positions and ‘policy networks’ to select moves. These deep neural networks are trained by a novel combination of supervised learning from human expert games, and reinforcement learning from games of self-play. Without any lookahead search, the neural networks play Go at the level of stateof-the-art Monte Carlo tree search programs that simulate thousands of random games of self-play. We also introduce a new search algorithm that combines Monte Carlo simulation with value and policy networks. Using this search algorithm, our program AlphaGo achieved a 99.8% winning rate against other Go programs, and defeated the human European Go champion by 5 games to 0. This is the first time that a computer program has defeated a human professional player in the full-sized game of Go, a feat previously thought to be at least a decade away.
---
paper_title: A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem

paper_content:

Financial portfolio management is the process of constant redistribution of a fund into different financial products. This paper presents a financial-model-free Reinforcement Learning framework to provide a deep machine learning solution to the portfolio management problem. The framework consists of the Ensemble of Identical Independent Evaluators (EIIE) topology, a Portfolio-Vector Memory (PVM), an Online Stochastic Batch Learning (OSBL) scheme, and a fully exploiting and explicit reward function. This framework is realized in three instants in this work with a Convolutional Neural Network (CNN), a basic Recurrent Neural Network (RNN), and a Long Short-Term Memory (LSTM). They are, along with a number of recently reviewed or published portfolio-selection strategies, examined in three back-test experiments with a trading period of 30 minutes in a cryptocurrency market. Cryptocurrencies are electronic and decentralized alternatives to government-issued money, with Bitcoin as the best-known example of a cryptocurrency. All three instances of the framework monopolize the top three positions in all experiments, outdistancing other compared trading algorithms. Although with a high commission rate of 0.25% in the backtests, the framework is able to achieve at least 4-fold returns in 50 days.
---
paper_title: Continuous control with deep reinforcement learning

paper_content:

Disclosed are methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a minibatch of experience tuples; and updating current values of the parameters of the actor neural network, comprising: for each experience tuple in the minibatch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; updating current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs; and updating the current values of the parameters of the actor neural network using the critic neural network.
---
paper_title: Reinforcement Learning: An Introduction

paper_content:

Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.
---
paper_title: Adversarial Deep Reinforcement Learning in Portfolio Management

paper_content:

In this paper, we implement three state-of-art continuous reinforcement learning algorithms, Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO) and Policy Gradient (PG)in portfolio management. All of them are widely-used in game playing and robot control. What's more, PPO has appealing theoretical propeties which is hopefully potential in portfolio management. We present the performances of them under different settings, including different learning rates, objective functions, feature combinations, in order to provide insights for parameters tuning, features selection and data preparation. We also conduct intensive experiments in China Stock market and show that PG is more desirable in financial market than DDPG and PPO, although both of them are more advanced. What's more, we propose a so called Adversarial Training method and show that it can greatly improve the training efficiency and significantly promote average daily return and sharpe ratio in back test. Based on this new modification, our experiments results show that our agent based on Policy Gradient can outperform UCRP.
---
paper_title: A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play

paper_content:

The game of chess is the longest-studied domain in the history of artificial intelligence. The strongest programs are based on a combination of sophisticated search techniques, domain-specific adaptations, and handcrafted evaluation functions that have been refined by human experts over several decades. By contrast, the AlphaGo Zero program recently achieved superhuman performance in the game of Go by reinforcement learning from self-play. In this paper, we generalize this approach into a single AlphaZero algorithm that can achieve superhuman performance in many challenging games. Starting from random play and given no domain knowledge except the game rules, AlphaZero convincingly defeated a world champion program in the games of chess and shogi (Japanese chess), as well as Go.
---
paper_title: The Pricing of Options and Corporate Liabilities

paper_content:

If options are correctly priced in the market, it should not be possible to make sure profits by creating portfolios of long and short positions in options and their underlying stocks. Using this principle, a theoretical valuation formula for options is derived. Since almost all corporate liabilities can be viewed as combinations of options, the formula and the analysis that led to it are also applicable to corporate liabilities such as common stock, corporate bonds, and warrants. In particular, the formula can be used to derive the discount that should be applied to a corporate bond because of the possibility of default.
---
paper_title: Risk and Asset Allocation

paper_content:

The statistics of asset allocation.- Univariate statistics.- Multivariate statistics.- Modeling the market.- Classical asset allocation.- Estimating the distribution of the market invariants.- Evaluating allocations.- Optimizing allocations.- Accounting for estimation risk.- Estimating the distribution of the market invariants.- Evaluating allocations.- Optimizing allocations.
---
paper_title: Reinforcement Learning: An Introduction

paper_content:

Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.
---
paper_title: Optimal Gambling Systems For Favorable Games

paper_content:

The following sections are included:IntroductionThe nature of Λ*The asymptotic time minimization problemAsymptotic magnitude problemProblems with finite goals in coin tossingREFERENCES
---
paper_title: The Kelly system maximizes median fortune

paper_content:

It is well known that the Kelly system of proportional betting, which maximizes the long-term geometric rate of growth of the gambler’s fortune, minimizes the expected time required to reach a specified goal. Less well known is the fact that it maximizes the median of the gambler’s fortune. This was pointed out by the author in a 1988 paper, but only under asymptotic assumptions that might cause one to question its applicability. Here we show that the result is true more generally, and argue that this is a desirable property of the Kelly system.
---
paper_title: Kelly Criterion for Multivariate Portfolios: A Model-Free Approach

paper_content:

The Kelly criterion is a money management principle that beats any other approach in many respects. In particular, it maximizes the expected growth rate and the median of the terminal wealth. However, until recently application of the Kelly criterion to multivariate portfolios has seen little analysis. We briefly introduce the Kelly criterion and then present its multivariate version based only on the first and the second moments of the asset excess returns. Additionally, we provide a simple numerical algorithm to manage virtually arbitrarily large portfolios according to so-called fractional Kelly strategies.
---
paper_title: Human-level control through deep reinforcement learning

paper_content:

The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.
---
paper_title: A Response to Professor Paul A. Samuelson's Objections to Kelly Capital Growth Investing

paper_content:

The Kelly Capital Growth Investment Strategy maximizes the expected utility of final wealth with a Bernoulli logarithmic utility function. In 1956 Kelly showed that static expected log maximization yields the maximum asymptotic long run growth. Good properties include minimizing the time to large asymptotic goals, maximizing the median, and being ahead on average after the first period. Bad properties include extremely large bets for short term favorable investment situations because the Arrow-Pratt risk aversion index is essentially zero. Paul Samuelson was a critic of this approach. His various points sent in letters to Ziemba are responded to. Samuelson's criticism is partially responsible for the current situation that most finance academics and investment professionals, except superior investors, do not recommend Kelly strategies. Samuelson's points are theoretically correct and sharpen the theory. They caution users of this approach to be careful and understand the true characteristics of these investments including ways to lower the investment exposure. His objections help us understand the theory better, but they do not detract from numerous valuable applications, some of which are briefly surveyed.
---
paper_title: Analysis of Kelly-optimal portfolios

paper_content:

We investigate the use of Kelly's strategy in the construction of an optimal portfolio of assets. For lognormally distributed asset returns, we derive approximate analytical results for the optimal investment fractions in various settings. We show that when mean returns and volatilities of the assets are small and there is no risk-free asset, the Kelly-optimal portfolio lies on Markowitz Efficient Frontier. Since in the investigated case the Kelly approach forbids short positions and borrowing, often only a small fraction of the available assets is included in the Kelly-optimal portfolio. This phenomenon, that we call condensation, is studied analytically in various model scenarios.
---
paper_title: Adaptive Markets: Financial Evolution at the Speed of Thought

paper_content:

A new, evolutionary explanation of markets and investor behavior ::: ::: Half of all Americans have money in the stock market, yet economists can't agree on whether investors and markets are rational and efficient, as modern financial theory assumes, or irrational and inefficient, as behavioral economists believe—and as financial bubbles, crashes, and crises suggest. This is one of the biggest debates in economics and the value or futility of investment management and financial regulation hang on the outcome. In this groundbreaking book, Andrew Lo cuts through this debate with a new framework, the Adaptive Markets Hypothesis, in which rationality and irrationality coexist. ::: ::: Drawing on psychology, evolutionary biology, neuroscience, artificial intelligence, and other fields, Adaptive Markets shows that the theory of market efficiency isn't wrong but merely incomplete. When markets are unstable, investors react instinctively, creating inefficiencies for others to exploit. Lo's new paradigm explains how financial evolution shapes behavior and markets at the speed of thought—a fact revealed by swings between stability and crisis, profit and loss, and innovation and regulation. ::: ::: A fascinating intellectual journey filled with compelling stories, Adaptive Markets starts with the origins of market efficiency and its failures, turns to the foundations of investor behavior, and concludes with practical implications—including how hedge funds have become the Galapagos Islands of finance, what really happened in the 2008 meltdown, and how we might avoid future crises. ::: ::: An ambitious new answer to fundamental questions in economics, Adaptive Markets is essential reading for anyone who wants to know how markets really work.
---
paper_title: On estimating the expected return on the market - an exploratory investigation

paper_content:

The expected market return is a number frequently required for the solution of many investment and corporate finance problems, but by comparison with other financial variables, there has been little research on estimating this expected return. Current practice for estimating the expected market return adds the historical average realized excess market returns to the Current observed interest rate. While this model explicitly reflects the dependence of the market return on the interest rate, it fails to account for the effect of changes in the level of market risk. Three models of equilibrium expected market returns which reflect this dependence are analyzed in this paper. Estimation procedures which incorporate the prior restriction that equilibrium expected excess returns on the market must be positive arc derived and applied to return data for the period 1926- 1978. The principal conclusions from this exploratory investigation are: (1) in estimating models of the expected market return. the non-negativity restriction of the expected excess return should be explicitly included as part of the specification; (2) estimators which use realized returns should be adjusted for heteroscedasticity.
---
paper_title: Adversarial Deep Reinforcement Learning in Portfolio Management

paper_content:

In this paper, we implement three state-of-art continuous reinforcement learning algorithms, Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO) and Policy Gradient (PG)in portfolio management. All of them are widely-used in game playing and robot control. What's more, PPO has appealing theoretical propeties which is hopefully potential in portfolio management. We present the performances of them under different settings, including different learning rates, objective functions, feature combinations, in order to provide insights for parameters tuning, features selection and data preparation. We also conduct intensive experiments in China Stock market and show that PG is more desirable in financial market than DDPG and PPO, although both of them are more advanced. What's more, we propose a so called Adversarial Training method and show that it can greatly improve the training efficiency and significantly promote average daily return and sharpe ratio in back test. Based on this new modification, our experiments results show that our agent based on Policy Gradient can outperform UCRP.
---
paper_title: Human-level control through deep reinforcement learning

paper_content:

The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.
---
paper_title: PERFORMANCE FUNCTIONS AND REINFORCEMENT LEARNING FOR TRADING SYSTEMS AND PORTFOLIOS

paper_content:

We propose to train trading systems and portfolios by optimizing objective functions that directly measure trading and investment performance. Rather than basing a trading system on forecasts or training via a supervised learning algorithm using labelled trading data, we train our systems using recurrent reinforcement learning (RRL) algorithms. The performance functions that we consider for reinforcement learning are profit or wealth, economic utility, the Sharpe ratio and our proposed differential Sharpe ratio. The trading and portfolio management systems require prior decisions as input in order to properly take into account the effects of transactions costs, market impact, and taxes. This temporal dependence on system state requires the use of reinforcement versions of standard recurrent learning algorithms. We present empirical results in controlled experiments that demonstrate the efficacy of some of our methods for optimizing trading systems and portfolios. For a long/short trader, we find that maximizing the differential Sharpe ratio yields more consistent results than maximizing profits, and that both methods outperform a trading system based on forecasts that minimize MSE. We find that portfolio traders trained to maximize the differential Sharpe ratio achieve better risk-adjusted returns than those trained to maximize profit. Finally, we provide simulation results for an S&P 500/TBill asset allocation system that demonstrate the presence of out-of-sample predictability in the monthly S&P 500 stock index for the 25 year period 1970 through 1994. Copyright © 1998 John Wiley & Sons, Ltd.
---
paper_title: Reinforcement learning in Portfolio Management and its interpretation

paper_content:

Since all machine learning methods commonly in use today are viewed ::: as black boxes, the goal of this paper is to make one of these meth- ::: ods transparent in the context of Portfolio management. I interpret ::: the strategies implied by reinforcement learning (RL) and relate them ::: to the strategies implied by academic portfolio advice with the help of ::: their classical portfolio (CP) management models. Because RL is ac- ::: tually approximate dynamic programming (DP), it is perfectly suited ::: for the volatile DP environment of portfolio management compared to ::: other machine learning methods in use. In terms of performance, this ::: RL method is able to: 1) achieve the same average terminal wealth of ::: 1.33, which is an increase of 33% in portfolio value over ve years, as ::: the CP model with a low risk-aversion 2) diminish the standard devia- ::: tion of the terminal wealth by 30% from 0.35 to 0.25 3) have a lower ::: turnover than the same CP model by three percent. This can mostly ::: be explained by the conservative investing of the reinforcement learning ::: method overall.
---
paper_title: Learning values across many orders of magnitude

paper_content:

Most learning algorithms are not invariant to the scale of the function that is being approximated. We propose to adaptively normalize the targets used in learning. This is useful in value-based reinforcement learning, where the magnitude of appropriate value approximations can change over time when we update the policy of behavior. Our main motivation is prior work on learning to play Atari games, where the rewards were all clipped to a predetermined range. This clipping facilitates learning across many different games with a single learning algorithm, but a clipped reward function can result in qualitatively different behavior. Using the adaptive normalization we can remove this domain-specific heuristic without diminishing overall performance.
---
paper_title: Model-Free reinforcement learning with continuous action in practice

paper_content:

Reinforcement learning methods are often considered as a potential solution to enable a robot to adapt to changes in real time to an unpredictable environment. However, with continuous action, only a few existing algorithms are practical for real-time learning. In such a setting, most effective methods have used a parameterized policy structure, often with a separate parameterized value function. The goal of this paper is to assess such actor-critic methods to form a fully specified practical algorithm. Our specific contributions include 1) developing the extension of existing incremental policy-gradient algorithms to use eligibility traces, 2) an empirical comparison of the resulting algorithms using continuous actions, 3) the evaluation of a gradient-scaling technique that can significantly improve performance. Finally, we apply our actor-critic algorithm to learn on a robotic platform with a fast sensorimotor cycle (10ms). Overall, these results constitute an important step towards practical real-time learning control with continuous action.
---
paper_title: Policy Gradient Methods for Reinforcement Learning with Function Approximation

paper_content:

Function approximation is essential to reinforcement learning, but the standard approach of approximating a value function and determining a policy from it has so far proven theoretically intractable. In this paper we explore an alternative approach in which the policy is explicitly represented by its own function approximator, independent of the value function, and is updated according to the gradient of expected reward with respect to the policy parameters. Williams's REINFORCE method and actor-critic methods are examples of this approach. Our main new result is to show that the gradient can be written in a form suitable for estimation from experience aided by an approximate action-value or advantage function. Using this result, we prove for the first time that a version of policy iteration with arbitrary differentiable function approximation is convergent to a locally optimal policy.
---
paper_title: Proximal Policy Optimization Algorithms

paper_content:

We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.
---
paper_title: Mastering the game of Go with deep neural networks and tree search

paper_content:

The game of Go has long been viewed as the most challenging of classic games for artificial intelligence owing to its enormous search space and the difficulty of evaluating board positions and moves. Here we introduce a new approach to computer Go that uses ‘value networks’ to evaluate board positions and ‘policy networks’ to select moves. These deep neural networks are trained by a novel combination of supervised learning from human expert games, and reinforcement learning from games of self-play. Without any lookahead search, the neural networks play Go at the level of stateof-the-art Monte Carlo tree search programs that simulate thousands of random games of self-play. We also introduce a new search algorithm that combines Monte Carlo simulation with value and policy networks. Using this search algorithm, our program AlphaGo achieved a 99.8% winning rate against other Go programs, and defeated the human European Go champion by 5 games to 0. This is the first time that a computer program has defeated a human professional player in the full-sized game of Go, a feat previously thought to be at least a decade away.
---
paper_title: A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem

paper_content:

Financial portfolio management is the process of constant redistribution of a fund into different financial products. This paper presents a financial-model-free Reinforcement Learning framework to provide a deep machine learning solution to the portfolio management problem. The framework consists of the Ensemble of Identical Independent Evaluators (EIIE) topology, a Portfolio-Vector Memory (PVM), an Online Stochastic Batch Learning (OSBL) scheme, and a fully exploiting and explicit reward function. This framework is realized in three instants in this work with a Convolutional Neural Network (CNN), a basic Recurrent Neural Network (RNN), and a Long Short-Term Memory (LSTM). They are, along with a number of recently reviewed or published portfolio-selection strategies, examined in three back-test experiments with a trading period of 30 minutes in a cryptocurrency market. Cryptocurrencies are electronic and decentralized alternatives to government-issued money, with Bitcoin as the best-known example of a cryptocurrency. All three instances of the framework monopolize the top three positions in all experiments, outdistancing other compared trading algorithms. Although with a high commission rate of 0.25% in the backtests, the framework is able to achieve at least 4-fold returns in 50 days.
---
paper_title: Continuous control with deep reinforcement learning

paper_content:

Disclosed are methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a minibatch of experience tuples; and updating current values of the parameters of the actor neural network, comprising: for each experience tuple in the minibatch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; updating current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs; and updating the current values of the parameters of the actor neural network using the critic neural network.
---
paper_title: Robust Log-Optimal Strategy with Reinforcement Learning

paper_content:

We proposed a new Portfolio Management method termed as Robust Log-Optimal Strategy (RLOS), which ameliorates the General Log-Optimal Strategy (GLOS) by approximating the traditional objective function with quadratic Taylor expansion. It avoids GLOS's complex CDF estimation process,hence resists the "Butterfly Effect" caused by estimation error. Besides,RLOS retains GLOS's profitability and the optimization problem involved in RLOS is computationally far more practical compared to GLOS. Further, we combine RLOS with Reinforcement Learning (RL) and propose the so-called Robust Log-Optimal Strategy with Reinforcement Learning (RLOSRL), where the RL agent receives the analyzed results from RLOS and observes the trading environment to make comprehensive investment decisions. The RLOSRL's performance is compared to some traditional strategies on several back tests, where we randomly choose a selection of constituent stocks of the CSI300 index as assets under management and the test results validate its profitability and stability.
---
paper_title: Reinforcement Learning: An Introduction

paper_content:

Reinforcement learning, one of the most active research areas in artificial intelligence, is a computational approach to learning whereby an agent tries to maximize the total amount of reward it receives when interacting with a complex, uncertain environment. In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. The book is divided into three parts. Part I defines the reinforcement learning problem in terms of Markov decision processes. Part II provides basic solution methods: dynamic programming, Monte Carlo methods, and temporal-difference learning. Part III presents a unified view of the solution methods and incorporates artificial neural networks, eligibility traces, and planning; the two final chapters present case studies and consider the future of reinforcement learning.
---
paper_title: Deterministic Policy Gradient Algorithms

paper_content:

In this paper we consider deterministic policy gradient algorithms for reinforcement learning with continuous actions. The deterministic policy gradient has a particularly appealing form: it is the expected gradient of the action-value function. This simple form means that the deterministic policy gradient can be estimated much more efficiently than the usual stochastic policy gradient. To ensure adequate exploration, we introduce an off-policy actor-critic algorithm that learns a deterministic target policy from an exploratory behaviour policy. We demonstrate that deterministic policy gradient algorithms can significantly outperform their stochastic counterparts in high-dimensional action spaces.
---
paper_title: A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play

paper_content:

The game of chess is the longest-studied domain in the history of artificial intelligence. The strongest programs are based on a combination of sophisticated search techniques, domain-specific adaptations, and handcrafted evaluation functions that have been refined by human experts over several decades. By contrast, the AlphaGo Zero program recently achieved superhuman performance in the game of Go by reinforcement learning from self-play. In this paper, we generalize this approach into a single AlphaZero algorithm that can achieve superhuman performance in many challenging games. Starting from random play and given no domain knowledge except the game rules, AlphaZero convincingly defeated a world champion program in the games of chess and shogi (Japanese chess), as well as Go.
---
paper_title: Trust Region Policy Optimization

paper_content:

In this article, we describe a method for optimizing control policies, with guaranteed monotonic improvement. By making several approximations to the theoretically-justified scheme, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is effective for optimizing large nonlinear policies such as neural networks. Our experiments demonstrate its robust performance on a wide variety of tasks: learning simulated robotic swimming, hopping, and walking gaits; and playing Atari games using images of the screen as input. Despite its approximations that deviate from the theory, TRPO tends to give monotonic improvement, with little tuning of hyperparameters.
---
paper_title: Deep Residual Learning for Image Recognition

paper_content:

Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.
---
paper_title: Proximal Policy Optimization Algorithms

paper_content:

We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the benefits of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time.
---
paper_title: A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem

paper_content:

Financial portfolio management is the process of constant redistribution of a fund into different financial products. This paper presents a financial-model-free Reinforcement Learning framework to provide a deep machine learning solution to the portfolio management problem. The framework consists of the Ensemble of Identical Independent Evaluators (EIIE) topology, a Portfolio-Vector Memory (PVM), an Online Stochastic Batch Learning (OSBL) scheme, and a fully exploiting and explicit reward function. This framework is realized in three instants in this work with a Convolutional Neural Network (CNN), a basic Recurrent Neural Network (RNN), and a Long Short-Term Memory (LSTM). They are, along with a number of recently reviewed or published portfolio-selection strategies, examined in three back-test experiments with a trading period of 30 minutes in a cryptocurrency market. Cryptocurrencies are electronic and decentralized alternatives to government-issued money, with Bitcoin as the best-known example of a cryptocurrency. All three instances of the framework monopolize the top three positions in all experiments, outdistancing other compared trading algorithms. Although with a high commission rate of 0.25% in the backtests, the framework is able to achieve at least 4-fold returns in 50 days.
---
paper_title: Adversarial Deep Reinforcement Learning in Portfolio Management

paper_content:

In this paper, we implement three state-of-art continuous reinforcement learning algorithms, Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO) and Policy Gradient (PG)in portfolio management. All of them are widely-used in game playing and robot control. What's more, PPO has appealing theoretical propeties which is hopefully potential in portfolio management. We present the performances of them under different settings, including different learning rates, objective functions, feature combinations, in order to provide insights for parameters tuning, features selection and data preparation. We also conduct intensive experiments in China Stock market and show that PG is more desirable in financial market than DDPG and PPO, although both of them are more advanced. What's more, we propose a so called Adversarial Training method and show that it can greatly improve the training efficiency and significantly promote average daily return and sharpe ratio in back test. Based on this new modification, our experiments results show that our agent based on Policy Gradient can outperform UCRP.
---
paper_title: Reinforcement learning in Portfolio Management and its interpretation

paper_content:

Since all machine learning methods commonly in use today are viewed ::: as black boxes, the goal of this paper is to make one of these meth- ::: ods transparent in the context of Portfolio management. I interpret ::: the strategies implied by reinforcement learning (RL) and relate them ::: to the strategies implied by academic portfolio advice with the help of ::: their classical portfolio (CP) management models. Because RL is ac- ::: tually approximate dynamic programming (DP), it is perfectly suited ::: for the volatile DP environment of portfolio management compared to ::: other machine learning methods in use. In terms of performance, this ::: RL method is able to: 1) achieve the same average terminal wealth of ::: 1.33, which is an increase of 33% in portfolio value over ve years, as ::: the CP model with a low risk-aversion 2) diminish the standard devia- ::: tion of the terminal wealth by 30% from 0.35 to 0.25 3) have a lower ::: turnover than the same CP model by three percent. This can mostly ::: be explained by the conservative investing of the reinforcement learning ::: method overall.
---
paper_title: A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem

paper_content:

Financial portfolio management is the process of constant redistribution of a fund into different financial products. This paper presents a financial-model-free Reinforcement Learning framework to provide a deep machine learning solution to the portfolio management problem. The framework consists of the Ensemble of Identical Independent Evaluators (EIIE) topology, a Portfolio-Vector Memory (PVM), an Online Stochastic Batch Learning (OSBL) scheme, and a fully exploiting and explicit reward function. This framework is realized in three instants in this work with a Convolutional Neural Network (CNN), a basic Recurrent Neural Network (RNN), and a Long Short-Term Memory (LSTM). They are, along with a number of recently reviewed or published portfolio-selection strategies, examined in three back-test experiments with a trading period of 30 minutes in a cryptocurrency market. Cryptocurrencies are electronic and decentralized alternatives to government-issued money, with Bitcoin as the best-known example of a cryptocurrency. All three instances of the framework monopolize the top three positions in all experiments, outdistancing other compared trading algorithms. Although with a high commission rate of 0.25% in the backtests, the framework is able to achieve at least 4-fold returns in 50 days.
---
paper_title: Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs

paper_content:

Generative Adversarial Networks (GANs) have shown remarkable success as a framework for training models to produce realistic-looking data. In this work, we propose a Recurrent GAN (RGAN) and Recurrent Conditional GAN (RCGAN) to produce realistic real-valued multi-dimensional time series, with an emphasis on their application to medical data. RGANs make use of recurrent neural networks (RNNs) in the generator and the discriminator. In the case of RCGANs, both of these RNNs are conditioned on auxiliary information. We demonstrate our models in a set of toy datasets, where we show visually and quantitatively (using sample likelihood and maximum mean discrepancy) that they can successfully generate realistic time-series. We also describe novel evaluation methods for GANs, where we generate a synthetic labelled training dataset, and evaluate on a real test set the performance of a model trained on the synthetic data, and vice-versa. We illustrate with these metrics that RCGANs can generate time-series data useful for supervised training, with only minor degradation in performance on real test data. This is demonstrated on digit classification from ‘serialised’ MNIST and by training an early warning system on a medical dataset of 17,000 patients from an intensive care unit. We further discuss and analyse the privacy concerns that may arise when using RCGANs to generate realistic synthetic medical time series data, and demonstrate results from differentially private training of the RCGAN.
---
paper_title: Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimization

paper_content:

Dynamic portfolio optimization is the process of sequentially allocating wealth to a collection of assets in some consecutive trading periods, based on investors' return-risk profile. Automating this process with machine learning remains a challenging problem. Here, we design a deep reinforcement learning (RL) architecture with an autonomous trading agent such that, investment decisions and actions are made periodically, based on a global objective, with autonomy. In particular, without relying on a purely model-free RL agent, we train our trading agent using a novel RL architecture consisting of an infused prediction module (IPM), a generative adversarial data augmentation module (DAM) and a behavior cloning module (BCM). Our model-based approach works with both on-policy or off-policy RL algorithms. We further design the back-testing and execution engine which interact with the RL agent in real time. Using historical {\em real} financial market data, we simulate trading with practical constraints, and demonstrate that our proposed model is robust, profitable and risk-sensitive, as compared to baseline trading strategies and model-free RL agents from prior work.
---
paper_title: Parallel WaveNet: Fast High-Fidelity Speech Synthesis

paper_content:

The recently-developed WaveNet architecture is the current state of the art in realistic speech synthesis, consistently rated as more natural sounding for many different languages than any previous system. However, because WaveNet relies on sequential generation of one audio sample at a time, it is poorly suited to today's massively parallel computers, and therefore hard to deploy in a real-time production setting. This paper introduces Probability Density Distillation, a new method for training a parallel feed-forward network from a trained WaveNet with no significant difference in quality. The resulting system is capable of generating high-fidelity speech samples at more than 20 times faster than real-time, and is deployed online by Google Assistant, including serving multiple English and Japanese voices.
---
paper_title: Adversarial Deep Reinforcement Learning in Portfolio Management

paper_content:

In this paper, we implement three state-of-art continuous reinforcement learning algorithms, Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO) and Policy Gradient (PG)in portfolio management. All of them are widely-used in game playing and robot control. What's more, PPO has appealing theoretical propeties which is hopefully potential in portfolio management. We present the performances of them under different settings, including different learning rates, objective functions, feature combinations, in order to provide insights for parameters tuning, features selection and data preparation. We also conduct intensive experiments in China Stock market and show that PG is more desirable in financial market than DDPG and PPO, although both of them are more advanced. What's more, we propose a so called Adversarial Training method and show that it can greatly improve the training efficiency and significantly promote average daily return and sharpe ratio in back test. Based on this new modification, our experiments results show that our agent based on Policy Gradient can outperform UCRP.
---

Title: Model-Free Reinforcement Learning for Financial Portfolios: A Brief Survey

Section 1: Introduction
Description 1: Introduce the area of model-free reinforcement learning in the context of financial portfolios and outline the motivation behind this survey.

Section 2: Portfolio Optimization
Description 2: Provide an overview of the three major paradigms in portfolio optimization (Mean Variance, Kelly Criterion, Risk Parity) and derive their equivalence.

Section 3: Mean Variance
Description 3: Discuss the details of Markowitz' Modern Portfolio Theory and briefly outline the computation methods and issues with optimization.

Section 4: Kelly Criterion
Description 4: Explain the principles of the Kelly Criterion and its application in portfolio optimization, as well as potential risks and recommendations for its use.

Section 5: Markowitz-Kelly Equivalence
Description 5: Show the mathematical equivalence between the Kelly Criterion and Markowitz' portfolio when certain conditions are met.

Section 6: Risk Parity
Description 6: Elaborate on the principles of Risk Parity, its advantages, and the different approaches within this paradigm.

Section 7: Markowitz-RP Equivalence
Description 7: Demonstrate the equivalence between Markowitz' portfolio optimization problem and Risk Parity under specific conditions.

Section 8: Model-Free Reinforcement Learning
Description 8: Discuss how model-free reinforcement learning can be applied to the portfolio optimization problem and provide an overview of its methodologies.

Section 9: Value-Based Methods
Description 9: Investigate value-based model-free RL methods like Q-Learning and their application to portfolio optimization, outlining various techniques and their drawbacks.

Section 10: Deep Deterministic Policy Gradient
Description 10: Examine the DDPG method, its application in portfolio optimization, and specific implementations from recent literature.

Section 11: Proximal Policy Optimization
Description 11: Introduce the PPO method, its theoretical foundation, and applications in portfolio management, highlighting recent experiments and findings.

Section 12: Conclusion and Discussion
Description 12: Summarize the survey findings, discuss unresolved issues, inherent challenges, and potential future directions in applying model-free RL to financial portfolios.