<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 TradingEnv-v0
Open AI 'Gym' for reinforcement-learning based trading algorithms
This gym implements a very simple trading environment for reinforcement learning.
The gym provides daily observations based on real market data pulled from Quandl on, by default, the SPY etf.  An episode is defined as 252 contiguous days sampled from the overall dataset.  Each day is one 'step' within the gym and for each step, the algo has a choice
Step1:  create the environment
This may take a moment as we are pulling historical data from quandl.
Step2:  the trading model
Each time step is a day.  Each episode is 252 trading days - a year.  Each day, we can choose to be short (0), flat (1) or long (2) the single instrument in our trading universe.
Let's run through a day and stay flat.
Step3:  Note that you are charged just for playing - to the tune of 1 basis point per day!
Rendering
For now, no rendering has been implemented for this gym, but with each step, the following datum are provided which you can easily graph and otherwise visualize as we see above with the NAV
Step4:  running the same strategy multiple times will likely yield different results as underlying data changes
Step5:  comparing the buyandhold and random traders
Step6:  Object of the game
From the above examples, we can see that buying and holding will, over the long run, give you the market return with low costs.
Randomly trading will instead destroy value rather quickly as costs overwhelm.
So, what does it mean to win the trading game?
For our purposes, we'll say that beating a buy & hold strategy, on average, over one hundred episodes will notch a win to the proud ai player.
To support this, the trading environment maintains the mkt_return which can be compared with the sim_return.
Note that the mkt_return is frictionless while the sim_return incurs both trading costs and the decay cost of 1 basis point per day, so overcoming the hurdle we've set here should be challenging.
Playing the game
Step7:  Results
Policy gradients beat the trading game!  That said, it doesn't work every time and it seems, looking at the charts below, as though it's a bit of a lucky thing.  But luck counts in the trading game as in life!
<ASSISTANT_TASK:>
Python Code:
import gym
import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
from matplotlib import interactive
interactive(True)
Explanation: TradingEnv-v0
Open AI 'Gym' for reinforcement-learning based trading algorithms
This gym implements a very simple trading environment for reinforcement learning.
The gym provides daily observations based on real market data pulled from Quandl on, by default, the SPY etf.  An episode is defined as 252 contiguous days sampled from the overall dataset.  Each day is one 'step' within the gym and for each step, the algo has a choice:
SHORT (0)
FLAT (1)
LONG  (2)
If you trade, you will be charged, by default, 10 BPS of the size of your trade.  Thus, going from short to long costs twice as much as going from short to/from flat.  Not trading also has a default cost of 1 BPS per step.  Nobody said it would be easy!
At the beginning of your episode, you are allocated 1 unit of cash.  This is your starting Net Asset Value (NAV). 
Beating the trading game
For our purposes, we'll say that beating a buy & hold strategy, on average, over one hundred episodes will notch a win to the proud ai player.  We'll illustrate exactly what that means below.
Let's look at some code using the environment
imports
End of explanation
env = gym.make('trading-v0')
#env.time_cost_bps = 0 # 
Explanation: create the environment
This may take a moment as we are pulling historical data from quandl.
End of explanation
observation = env.reset()
done = False
navs = []
while not done:
    action = 1 # stay flat
    observation, reward, done, info = env.step(action)
    navs.append(info['nav'])
    if done:
        print 'Annualized return: ',navs[len(navs)-1]-1
        pd.DataFrame(navs).plot()
Explanation: the trading model
Each time step is a day.  Each episode is 252 trading days - a year.  Each day, we can choose to be short (0), flat (1) or long (2) the single instrument in our trading universe.
Let's run through a day and stay flat.
End of explanation
import trading_env as te
stayflat     = lambda o,e: 1   # stand pat
buyandhold   = lambda o,e: 2   # buy on day #1 and hold
randomtrader = lambda o,e: e.action_space.sample() # retail trader
# to run singly, we call run_strat.  we are returned a dataframe containing 
#  all steps in the sim.
bhdf = env.run_strat(buyandhold)
print bhdf.head()
# we can easily plot our nav in time:
bhdf.bod_nav.plot(title='buy & hold nav')
Explanation: Note that you are charged just for playing - to the tune of 1 basis point per day!
Rendering
For now, no rendering has been implemented for this gym, but with each step, the following datum are provided which you can easily graph and otherwise visualize as we see above with the NAV:
pnl - how much did we make or lose between yesterday and today?
costs  - how much did we pay in costs today
nav    - our current nav
utility methods: running strategies once or repeatedly
Although the gym can be 'exercised' directly as seen above, we've also written utility methods which allow for the running of a strategy once or over many episodes, facilitating training or other sorts of analysis.
To utilize these methods, strategies should be exposed as a function or lambda with the following signature:
Action a = strategy( observation, environment )
Below, we define some simple strategies and look briefly at their behavior to better understand the trading gym.
End of explanation
env.run_strat(buyandhold).bod_nav.plot(title='same strat, different results')
env.run_strat(buyandhold).bod_nav.plot()
env.run_strat(buyandhold).bod_nav.plot()
Explanation: running the same strategy multiple times will likely yield different results as underlying data changes
End of explanation
# running a strategy multiple times should yield insights 
#   into its expected behavior or give it oppty to learn
bhdf = env.run_strats(buyandhold,100)
rdf = env.run_strats(randomtrader,100)
comparo = pd.DataFrame({'buyhold':bhdf.mean(),
                        'random': rdf.mean()})
comparo
Explanation: comparing the buyandhold and random traders
End of explanation
import tensorflow as tf
import policy_gradient
# create the tf session
sess = tf.InteractiveSession()
# create policygradient
pg = policy_gradient.PolicyGradient(sess, obs_dim=5, num_actions=3, learning_rate=1e-2 )
# and now let's train it and evaluate its progress.  NB: this could take some time...
df,sf = pg.train_model( env,episodes=25001, log_freq=100)#, load_model=True)
Explanation: Object of the game
From the above examples, we can see that buying and holding will, over the long run, give you the market return with low costs.
Randomly trading will instead destroy value rather quickly as costs overwhelm.
So, what does it mean to win the trading game?
For our purposes, we'll say that beating a buy & hold strategy, on average, over one hundred episodes will notch a win to the proud ai player.
To support this, the trading environment maintains the mkt_return which can be compared with the sim_return.
Note that the mkt_return is frictionless while the sim_return incurs both trading costs and the decay cost of 1 basis point per day, so overcoming the hurdle we've set here should be challenging.
Playing the game: purloined policy gradients
I've taken and adapted (see code for details) a policy gradient implementation based on tensorflow to try to play the single-instrument trading game.  Let's see how it does.
End of explanation
sf['net'] = sf.simror - sf.mktror
#sf.net.plot()
sf.net.expanding().mean().plot()
sf.net.rolling(100).mean().plot()
sf.net.rolling(100).mean().tail()
Explanation: Results
Policy gradients beat the trading game!  That said, it doesn't work every time and it seems, looking at the charts below, as though it's a bit of a lucky thing.  But luck counts in the trading game as in life!
End of explanation
<END_TASK>