<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 DQNs on GCP
Reinforcement Learning (RL) Agents can be quite fickle. This is because the environment for an Agent is different than that of Supervised and Unsupervised algorithms.
| Supervised / Unsupervised | Reinforcement Learning |
| ----------- | ----------- |
| Data is previously gathered | Data needs to be simulated |
| Big Data
Step1:  The above command sends a hyperparameter tuning job to the Google Cloud AI Platform. It's a service that sets up scaling distributed training so data scientists and machine learning engineers do not have to worry about technical infrastructure. Usually, it automatically selects the container environment, but we're going to take advantage of a feature to specify our own environment with Docker. Not only will this allow us to install our game environment to be deployed to the cloud, but it will also significantly speed up hyperparameter tuning time as each worker can skip the library installation steps.
The <a href="Dockerfile">Dockerfile</a> in this directory shows the steps taken to build this environment. First, we copy from a Google Deep Learning Container which already has Google Cloud Libraries installed. Then, we install our other desired modules and libraries. ffmpeg, xvfb, and python-opengl are needed in order to get video output from the server. Machines on the cloud don't typically have a display (why would they need one?), so we'll make a virtual display of our own.
After we copy our code, we tell the container to be configured as an executable so we can pass our hyperparameter tuning flags to it with the ENTRYPOINT command. In order to set up our virtual display, we can use the xvfb-run command. Unfortunately, Docker strips quotes from specified commands in ENTRYPOINT, so we'll make a super simple shell script, <a href="train_model.sh">train_model.sh</a>, to specify our virtual display parameters. The "@" parameter is used to pass the flags called against the container to our python module, trainer.trainer.
CartPole-v0
So what is the game we'll be solving for? We'll be playing with AI Gym's CartPole Environment. As MNIST is the "Hello World" of image classification, CartPole is the "Hello World" of Deep Q Networks. Let's install OpenAI Gym and play with the game ourselves!
Step2:  Note
Step3:  The reset method will restart the environment and return a starting state.
The step method takes an action, applies it to the environment and returns a new state. Each step returns a new state, the transition reward, whether the game is over or not, and game specific information. For CartPole, there is no extra info, so it returns a blank dictionary.
Step4:  Run the cell below repeatedly until the game is over, changing the action to push the cart left (0) or right (1). The game is considered "won" when the pole can stay up for an average of steps 195 over 100 games. How far can you get? An agent acting randomly can only survive about 10 steps.
Step5:  We can make our own policy and create a loop to play through an episode (one full simulation) of the game. Below, actions are generated to alternate between pushing the cart left and right. The code is very similar to how our agents will be interacting with the game environment.
Step7:  It's a challenge to get to 200! We could repeatedly experiment to find the best heuristics to beat the game, or we could leave all that work to the robot. Let's create an intelligence to figure this out for us.
The Theory Behind Deep Q Networks
The fundamental principle behind RL is we have two entities
Step11:  Notice any other atypical aspects of this network?
Here, we take in both state and actions as inputs to our network. The states are fed in as normal, but the actions are used to "mask" the output. This is actually used for faster training, as we'd only want to update the nodes correspnding to the action that we simulated.
The Bellman Equation actually isn't in the network. That's because this is only the "brain" of our agent. As an intelligence, it has much more! Before we get to how exactly the agent learns, let's looks at the other aspects of its body
Step12:  Let's make a fake buffer and play around with it! We'll add the memory into our game play code to start collecting experiences.
Step13:  Now, let's sample the memory by running the cell below multiple times. It's different each call, and that's on purpose. Just like with other neural networks, it's important to randomly sample so that our agent can learn from many different situations.
The use of a memory buffer is called Experience Replay. The above technique of a uniform random sample is a quick and computationally efficient way to get the job done, but RL researchers often look into other sampling methods. For instance, maybe there's a way to weight memories based on their rarity or loss when the agent learns with it.
Step17:  But before the agent has any memories and has learned anything, how is it supposed to act? That comes down to Exploration vs Exploitation. The trouble is that in order to learn, risks with the unknown need to be made. There's no right answer, but there is a popular answer. We'll start by acting randomly, and over time, we will slowly decay our chance to act randomly.
Below is a partial version of the agent.
Step18:  Let's define the agent and get a starting state to see how it would act without any training.
Step19:  Run the cell below multiple times. Since we're decaying the random action rate after every action, it's only a matter a time before the agent exploits more than it explores.
Step21:  Memories, a brain, and a healthy dose of curiosity. We finally have all the ingredient for our agent to learn. After all, as the Scarecrow from the Wizard of Oz said
Step22:  Nice! We finally have an intelligence that can walk and talk and... well ok, this intelligence is too simple to be able to do those things, but maybe it can learn to push a cart with a pole on it. Let's update our training loop to use our new agent.
Run the below cell over and over up to ten times to train the agent.
Step25:  Hypertuning
Chances are, at this point, the agent is having a tough time learning. Why is that? Well, remember that hyperparameter tuning job we kicked off at the start of this notebook?
The are many parameters that need adjusting with our agent. Let's recap
<ASSISTANT_TASK:>
Python Code:
%%bash
BUCKET=<your-bucket-here> # Change to your bucket name
JOB_NAME=dqn_on_gcp_$(date -u +%y%m%d_%H%M%S)
REGION='us-central1' # Change to your bucket region
IMAGE_URI=gcr.io/qwiklabs-resources/rl-qwikstart/dqn_on_gcp@sha256:326427527d07f30a0486ee05377d120cac1b9be8850b05f138fc9b53ac1dd2dc
gcloud ai-platform jobs submit training $JOB_NAME \
    --staging-bucket=gs://$BUCKET \
    --region=$REGION \
    --master-image-uri=$IMAGE_URI \
    --scale-tier=BASIC_GPU \
    --job-dir=gs://$BUCKET/$JOB_NAME \
    --config=hyperparam.yaml
Explanation: DQNs on GCP
Reinforcement Learning (RL) Agents can be quite fickle. This is because the environment for an Agent is different than that of Supervised and Unsupervised algorithms.
| Supervised / Unsupervised | Reinforcement Learning |
| ----------- | ----------- |
| Data is previously gathered | Data needs to be simulated |
| Big Data: Many examples covering many siutations | Sparse Data: Agent trades off between exploring and exploiting | 
| The environment is assumed static | The environment may change in response to the agent |
Because of this, hyperparameter tuning is even more crucial in RL as it not only impacts the training of the agent's neural network, but it also impacts how the data is gathered through simulation.
Setup
Hypertuning takes some time, and in this case, it can take anywhere between 10 - 30 minutes. If this hasn't been done already, run the cell below to kick off the training job now. We'll step through what the code is doing while our agents learn.
End of explanation
!python3 -m pip freeze | grep gym || python3 -m pip install --user gym==0.12.5
!python3 -m pip freeze | grep 'tensorflow==2\|tensorflow-gpu==2' || \
    python3 -m pip install --user tensorflow==2
Explanation: The above command sends a hyperparameter tuning job to the Google Cloud AI Platform. It's a service that sets up scaling distributed training so data scientists and machine learning engineers do not have to worry about technical infrastructure. Usually, it automatically selects the container environment, but we're going to take advantage of a feature to specify our own environment with Docker. Not only will this allow us to install our game environment to be deployed to the cloud, but it will also significantly speed up hyperparameter tuning time as each worker can skip the library installation steps.
The <a href="Dockerfile">Dockerfile</a> in this directory shows the steps taken to build this environment. First, we copy from a Google Deep Learning Container which already has Google Cloud Libraries installed. Then, we install our other desired modules and libraries. ffmpeg, xvfb, and python-opengl are needed in order to get video output from the server. Machines on the cloud don't typically have a display (why would they need one?), so we'll make a virtual display of our own.
After we copy our code, we tell the container to be configured as an executable so we can pass our hyperparameter tuning flags to it with the ENTRYPOINT command. In order to set up our virtual display, we can use the xvfb-run command. Unfortunately, Docker strips quotes from specified commands in ENTRYPOINT, so we'll make a super simple shell script, <a href="train_model.sh">train_model.sh</a>, to specify our virtual display parameters. The "@" parameter is used to pass the flags called against the container to our python module, trainer.trainer.
CartPole-v0
So what is the game we'll be solving for? We'll be playing with AI Gym's CartPole Environment. As MNIST is the "Hello World" of image classification, CartPole is the "Hello World" of Deep Q Networks. Let's install OpenAI Gym and play with the game ourselves!
End of explanation
from collections import deque
import random
import gym
import numpy as np
import tensorflow as tf
from tensorflow.keras import layers, models
env = gym.make('CartPole-v0')
print("The observation space is", env.observation_space)
print("The observation dimensions are", env.observation_space.shape)
print("The action space is", env.action_space)
print("The number of possible actions is", env.action_space.n)
Explanation: Note: Restart the kernel if the above libraries needed to be installed
The gym library hosts a number of different gaming environments that our agents (and us humans) can play around in. To make an environment, we simply need to pass it what game we'd like to play with the make method.
This will create an environment object with a number of useful methods and properties.
* The observation_space parameter is the structure of observations about the environment.
  - Each "state" or snapshot or our environment will follow this structure
* The action_space parameter is the possible actions the agent can take
So for example, with CartPole, there are 4 observation dimensions which represent [Cart Position, Cart Velocity, Pole Angle, Pole Velocity At Tip]. For the actions, there are 2 possible actions to take: 0 pushes the cart to the left, and 1 pushes the cart to the right. More detail is described in the game's code here.
End of explanation
def print_state(state, step, reward=None):
    format_string = 'Step {0} - Cart X: {1:.3f}, Cart V: {2:.3f}, Pole A: {3:.3f}, Pole V:{4:.3f}, Reward:{5}'
    print(format_string.format(step, *tuple(state), reward))
state = env.reset()
step = 0
print_state(state, step)
action = 0
state_prime, reward, done, info = env.step(action)
step += 1
print_state(state_prime, step, reward)
print("The game is over." if done else "The game can continue.")
print("Info:", info)
Explanation: The reset method will restart the environment and return a starting state.
The step method takes an action, applies it to the environment and returns a new state. Each step returns a new state, the transition reward, whether the game is over or not, and game specific information. For CartPole, there is no extra info, so it returns a blank dictionary.
End of explanation
action = 1  # Change me: 0 Left, 1 Right
state_prime, reward, done, info = env.step(action)
step += 1
print_state(state_prime, step, reward)
print("The game is over." if done else "The game can continue.")
Explanation: Run the cell below repeatedly until the game is over, changing the action to push the cart left (0) or right (1). The game is considered "won" when the pole can stay up for an average of steps 195 over 100 games. How far can you get? An agent acting randomly can only survive about 10 steps.
End of explanation
# [0, 1, 0, 1, 0, 1, ...]
actions = [x % 2 for x in range(200)]
state = env.reset()
step = 0
episode_reward = 0
done = False
while not done and step < len(actions):
    action = actions[step]  # In the future, our agents will define this.
    state_prime, reward, done, info = env.step(action)
    episode_reward += reward
    step += 1
    state = state_prime
    print_state(state, step, reward)
end_statement = "Game over!" if done else "Ran out of actions!"
print(end_statement, "Score =", episode_reward)
Explanation: We can make our own policy and create a loop to play through an episode (one full simulation) of the game. Below, actions are generated to alternate between pushing the cart left and right. The code is very similar to how our agents will be interacting with the game environment.
End of explanation
def deep_q_network(
        state_shape, action_size, learning_rate, hidden_neurons):
    Creates a Deep Q Network to emulate Q-learning.
    Creates a two hidden-layer Deep Q Network. Similar to a typical nueral
    network, the loss function is altered to reduce the difference between
    predicted Q-values and Target Q-values.
    Args:
        space_shape: a tuple of ints representing the observation space.
        action_size (int): the number of possible actions.
        learning_rate (float): the nueral network's learning rate.
        hidden_neurons (int): the number of neurons to use per hidden
            layer.
    
    state_input = layers.Input(state_shape, name='frames')
    actions_input = layers.Input((action_size,), name='mask')
    hidden_1 = layers.Dense(hidden_neurons, activation='relu')(state_input)
    hidden_2 = layers.Dense(hidden_neurons, activation='relu')(hidden_1)
    q_values = layers.Dense(action_size)(hidden_2)
    masked_q_values = layers.Multiply()([q_values, actions_input])
    model = models.Model(
        inputs=[state_input, actions_input], outputs=masked_q_values)
    optimizer = tf.keras.optimizers.RMSprop(lr=learning_rate)
    model.compile(loss='mse', optimizer=optimizer)
    return model
Explanation: It's a challenge to get to 200! We could repeatedly experiment to find the best heuristics to beat the game, or we could leave all that work to the robot. Let's create an intelligence to figure this out for us.
The Theory Behind Deep Q Networks
The fundamental principle behind RL is we have two entities: the agent and the environment. The agent takes state and reward information about the envionment and chooses an action. The environment takes that action and will change to be in a new state.
<img src="images/agent_and_environment.jpg" width="476" height="260">
RL assumes that the environment follows a Markov Decision Process (MDP). That means the state is dependent partially on the agent's actions, and partially on chance. MDPs can be represented by a graph, with states and actions as nodes, and rewards and path probabilities on the edges.
<img src="images/mdp.jpg" width="471" height="243">
So what would be the best path through the graph above? Or perhaps a more difficult question, what would be our expected winnings if we played optimally? The probability introduced in this problem has inspired multiple strategies over the years, but all of them boil down to the idea of discounted future rewards.
Would you rather have $100 now or $105 a year from now? With inflation, there's no definitive answer, but each of us has a threshold that we use to determine the value of something now versus the value of something later. In psychology, this is called Delayed Gratification. Richard E. Bellman expressed this theory in an equation widely used in RL called the Bellman Equation. Let's introduce some vocab to better define it.
| Symbol | Name | Definition | Example |
| - | - | - | - |
| | agent | An entity that can act and transition between states | Us when we play CartPole |
| s | state | The environmental parameters describing where the agent is | The position of the cart and angle of the pole |
| a | action | What the agent can do within a state | Pushing the cart left or right |
| t | time / step | One transition between states | One push of the cart |
|| episode | One full simulation run | From the start of the game to game over |
| v, V(s) | value | How much a state is worth | V(last state dropping the pole) = 0
| r, R(s, a) | reward | Value gained or lost transitioning between states through an action | R(keeping the pole up) = 1 |
| γ | gamma | How much to value a current state based on a future state | Coming up soon |
| 𝜋, 𝜋(s) | policy |The recommended action to the agent based on the current state | π(in trouble) = honesty |
Bellman realized this: The value of our current state should the discounted value of the next state the agent will be in plus any rewards picked up along the way, given the agent takes the best action to maximize this.
Using all the symbols from above, we get:
<img src="images/bellman_equation.jpg" width="260" height="50">
However, this is assuming we know all the states, their corresponding actions, and their rewards. If we don't know this in advance, we can explore and simulate this equation with what is called the Q equation:
<img style="background-color:white;" src="https://wikimedia.org/api/rest_v1/media/math/render/svg/47fa1e5cf8cf75996a777c11c7b9445dc96d4637">
Here, the value function is replaced with the Q value, which is a function of a state and action. The learning rate is how much we want to change our old Q value with new information found during simulation. Visually, this results in a Q-table, where rows are the states, actions are the columns, and each cell is the value found through simulation.
|| Meal | Snack | Wait |
|-|-|-|-|
| Hangry | 1 | .5 | -1 |
| Hungry | .5 | 1 | 0 |
| Full | -1 | -.5 | 1.5 |
So this is cool and all, but how exactly does this fit in with CartPole? Here, MDPs are discrete states. CartPole has multidimensional states on a continuous scale. This is where neural networks save the day! Rather than categorize each state, we can feed state properties into our network. By having the same number of output nodes as possible actions, our network can be used to predict the value of the next state given the current state and action.
Building the Agent
These networks can be configured with the same architectures and tools as other problems, such as CNNs. However, the one gotcha is that uses a specialized loss function. We'll instead be using the derivative of the Bellman Equation. Let's go ahead and define our model function as it is in trainer/model.py
End of explanation
class Memory():
    Sets up a memory replay buffer for a Deep Q Network.
    A simple memory buffer for a DQN. This one randomly selects state
    transitions with uniform probability, but research has gone into
    other methods. For instance, a weight could be given to each memory
    depending on how big of a difference there is between predicted Q values
    and target Q values.
    Args:
        memory_size (int): How many elements to hold in the memory buffer.
        batch_size (int): The number of elements to include in a replay batch.
        gamma (float): The "discount rate" used to assess Q values.
    
    def __init__(self, memory_size, batch_size, gamma):
        self.buffer = deque(maxlen=memory_size)
        self.batch_size = batch_size
        self.gamma = gamma
    def add(self, experience):
        Adds an experience into the memory buffer.
        Args:
            experience: a (state, action, reward, state_prime, done) tuple.
        
        self.buffer.append(experience)
    def sample(self):
        Uniformally selects from the replay memory buffer.
        Uniformally and randomly selects experiences to train the nueral
        network on. Transposes the experiences to allow batch math on
        the experience components.
        Returns:
            (list): A list of lists with structure [
                [states], [actions], [rewards], [state_primes], [dones]
            ]
        
        buffer_size = len(self.buffer)
        index = np.random.choice(
            np.arange(buffer_size), size=self.batch_size, replace=False)
        # Columns have different data types, so numpy array would be awkward.
        batch = np.array([self.buffer[i] for i in index]).T.tolist()
        states_mb = tf.convert_to_tensor(np.array(batch[0], dtype=np.float32))
        actions_mb = np.array(batch[1], dtype=np.int8)
        rewards_mb = np.array(batch[2], dtype=np.float32)
        states_prime_mb = np.array(batch[3], dtype=np.float32)
        dones_mb = batch[4]
        return states_mb, actions_mb, rewards_mb, states_prime_mb, dones_mb
Explanation: Notice any other atypical aspects of this network?
Here, we take in both state and actions as inputs to our network. The states are fed in as normal, but the actions are used to "mask" the output. This is actually used for faster training, as we'd only want to update the nodes correspnding to the action that we simulated.
The Bellman Equation actually isn't in the network. That's because this is only the "brain" of our agent. As an intelligence, it has much more! Before we get to how exactly the agent learns, let's looks at the other aspects of its body: "Memory" and "Exploration".
Just like other neural network algorithms, we need data to train on. However, this data is the result of our simulations, not something previously stored in a table. Thus, we're going to give our agent a memory where we can store state - action - new state transitions to learn on.
Each time the agent takes a step in gym, we'll save (state, action, reward, state_prime, done) to our buffer, which is defined like so.
End of explanation
test_memory_size = 20
test_batch_size = 4
test_gamma = .9  # Unused here. For learning.
test_memory = Memory(test_memory_size, test_batch_size, test_gamma)
actions = [x % 2 for x in range(200)]
state = env.reset()
step = 0
episode_reward = 0
done = False
while not done and step < len(actions):
    action = actions[step]  # In the future, our agents will define this.
    state_prime, reward, done, info = env.step(action)
    episode_reward += reward
    test_memory.add((state, action, reward, state_prime, done)) # New line here
    step += 1
    state = state_prime
    print_state(state, step, reward)
end_statement = "Game over!" if done else "Ran out of actions!"
print(end_statement, "Score =", episode_reward)
Explanation: Let's make a fake buffer and play around with it! We'll add the memory into our game play code to start collecting experiences.
End of explanation
test_memory.sample()
Explanation: Now, let's sample the memory by running the cell below multiple times. It's different each call, and that's on purpose. Just like with other neural networks, it's important to randomly sample so that our agent can learn from many different situations.
The use of a memory buffer is called Experience Replay. The above technique of a uniform random sample is a quick and computationally efficient way to get the job done, but RL researchers often look into other sampling methods. For instance, maybe there's a way to weight memories based on their rarity or loss when the agent learns with it.
End of explanation
class Partial_Agent():
    Sets up a reinforcement learning agent to play in a game environment.
    def __init__(self, network, memory, epsilon_decay, action_size):
        Initializes the agent with DQN and memory sub-classes.
        Args:
            network: A neural network created from deep_q_network().
            memory: A Memory class object.
            epsilon_decay (float): The rate at which to decay random actions.
            action_size (int): The number of possible actions to take.
        
        self.network = network
        self.action_size = action_size
        self.memory = memory
        self.epsilon = 1  # The chance to take a random action.
        self.epsilon_decay = epsilon_decay
    def act(self, state, training=False):
        Selects an action for the agent to take given a game state.
        Args:
            state (list of numbers): The state of the environment to act on.
            traning (bool): True if the agent is training.
        Returns:
            (int) The index of the action to take.
        
        if training:
            # Random actions until enough simulations to train the model.
            if len(self.memory.buffer) >= self.memory.batch_size:
                self.epsilon *= self.epsilon_decay
            if self.epsilon > np.random.rand():
                print("Exploration!")
                return random.randint(0, self.action_size-1)
        # If not acting randomly, take action with highest predicted value.
        print("Exploitation!")
        state_batch = np.expand_dims(state, axis=0)
        predict_mask = np.ones((1, self.action_size,))
        action_qs = self.network.predict([state_batch, predict_mask])
        return np.argmax(action_qs[0])
Explanation: But before the agent has any memories and has learned anything, how is it supposed to act? That comes down to Exploration vs Exploitation. The trouble is that in order to learn, risks with the unknown need to be made. There's no right answer, but there is a popular answer. We'll start by acting randomly, and over time, we will slowly decay our chance to act randomly.
Below is a partial version of the agent.
End of explanation
state = env.reset()
# Define "brain"
space_shape = env.observation_space.shape
action_size = env.action_space.n
# Feel free to play with these
test_learning_rate = .2
test_hidden_neurons = 10
test_epsilon_decay = .95
test_network = deep_q_network(
    space_shape, action_size, test_learning_rate, test_hidden_neurons)
test_agent = Partial_Agent(
    test_network, test_memory, test_epsilon_decay, action_size)
Explanation: Let's define the agent and get a starting state to see how it would act without any training.
End of explanation
action = test_agent.act(state, training=True)
print("Push Right" if action else "Push Left")
Explanation: Run the cell below multiple times. Since we're decaying the random action rate after every action, it's only a matter a time before the agent exploits more than it explores.
End of explanation
def learn(self):
    Trains the Deep Q Network based on stored experiences.
    batch_size = self.memory.batch_size
    if len(self.memory.buffer) < batch_size:
        return None
    # Obtain random mini-batch from memory.
    state_mb, action_mb, reward_mb, next_state_mb, done_mb = (
        self.memory.sample())
    # Get Q values for next_state.
    predict_mask = np.ones(action_mb.shape + (self.action_size,))
    next_q_mb = self.network.predict([next_state_mb, predict_mask])
    next_q_mb = tf.math.reduce_max(next_q_mb, axis=1)
    # Apply the Bellman Equation
    target_qs = (next_q_mb * self.memory.gamma) + reward_mb
    target_qs = tf.where(done_mb, reward_mb, target_qs)
    # Match training batch to network output:
    # target_q where action taken, 0 otherwise.
    action_mb = tf.convert_to_tensor(action_mb, dtype=tf.int32)
    action_hot = tf.one_hot(action_mb, self.action_size)
    target_mask = tf.multiply(tf.expand_dims(target_qs, -1), action_hot)
    return self.network.train_on_batch(
        [state_mb, action_hot], target_mask, reset_metrics=False
    )
Partial_Agent.learn = learn
test_agent = Partial_Agent(
    test_network, test_memory, test_epsilon_decay, action_size)
Explanation: Memories, a brain, and a healthy dose of curiosity. We finally have all the ingredient for our agent to learn. After all, as the Scarecrow from the Wizard of Oz said:
"Everything in life is unusual until you get accustomed to it."
~L. Frank Baum
Below is the code used by our agent to learn, where the Bellman Equation at last makes an appearance. We'll run through the following steps.
Pull a batch from memory
Get the Q value (the output of the neural network) based on the memory's ending state
Assume the Q value of the action with the highest Q value (test all actions)
Update these Q values with the Bellman Equation
target_qs = (next_q_mb * self.memory.gamma) + reward_mb
If the state is the end of the game, set the target_q to the reward for entering the final state.
Reshape the target_qs to match the networks output
Only learn on the memory's corresponding action by setting all action nodes to zero besides the action node taken.
Fit Target Qs as the label to our model against the memory's starting state and action as the inputs.
End of explanation
state = env.reset()
step = 0
episode_reward = 0
done = False
while not done:
    action = test_agent.act(state, training=True)
    state_prime, reward, done, info = env.step(action)
    episode_reward += reward
    test_agent.memory.add((state, action, reward, state_prime, done)) # New line here
    step += 1
    state = state_prime
    print_state(state, step, reward)
print(test_agent.learn())
print("Game over! Score =", episode_reward)
Explanation: Nice! We finally have an intelligence that can walk and talk and... well ok, this intelligence is too simple to be able to do those things, but maybe it can learn to push a cart with a pole on it. Let's update our training loop to use our new agent.
Run the below cell over and over up to ten times to train the agent.
End of explanation
def _parse_arguments(argv):
    Parses command-line arguments.
    parser = argparse.ArgumentParser()
    parser.add_argument(
        '--game',
        help='Which open ai gym game to play',
        type=str,
        default='CartPole-v0')
    parser.add_argument(
        '--episodes',
        help='The number of episodes to simulate',
        type=int,
        default=200)
    parser.add_argument(
        '--learning_rate',
        help='Learning rate for the nueral network',
        type=float,
        default=0.2)
    parser.add_argument(
        '--hidden_neurons',
        help='The number of nuerons to use per layer',
        type=int,
        default=30)
    parser.add_argument(
        '--gamma',
        help='The gamma or "discount" factor to discount future states',
        type=float,
        default=0.5)
    parser.add_argument(
        '--explore_decay',
        help='The rate at which to decay the probability of a random action',
        type=float,
        default=0.1)
    parser.add_argument(
        '--memory_size',
        help='Size of the memory buffer',
        type=int,
        default=100000)
    parser.add_argument(
        '--memory_batch_size',
        help='The amount of memories to sample from the buffer while training',
        type=int,
        default=8)
    parser.add_argument(
        '--job-dir',
        help='Directory where to save the given model',
        type=str,
        default='models/')
    parser.add_argument(
        '--print_rate',
        help='How often to print the score, 0 if never',
        type=int,
        default=0)
    parser.add_argument(
        '--eval_rate',
        help=While training, perform an on-policy simulation and record
        metrics to tensorboard every <record_rate> steps, 0 if never. Use
        higher values to avoid hyperparameter tuning "too many metrics"
        error,
        type=int,
        default=20)
    return parser.parse_known_args(argv)
Explanation: Hypertuning
Chances are, at this point, the agent is having a tough time learning. Why is that? Well, remember that hyperparameter tuning job we kicked off at the start of this notebook?
The are many parameters that need adjusting with our agent. Let's recap:
* The number of episodes or full runs of the game to train on
* The neural networks learning_rate
* The number of hidden_neurons to use in our network
* gamma, or how much we want to discount the future value of states
* How quickly we want to switch from explore to exploit with explore_decay
* The size of the memory buffer, memory_size
* The number of memories to pull from the buffer when training, memory_batch_size
These all have been added as flags to pass to the model in trainer/trainer.py's _parse_arguments method. For the most part, trainer/trainer.py follows the structure of the training loop that we have above, but it does have a few extra bells and whistles, like a hook into TensorBoard and video output.
End of explanation
<END_TASK>