<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Getting Started Guide
Table of Contents
Using Coach from the Command Line
Using Coach as a Library
Preset based - using CoachInterface
Training a preset
Running each training or inference iteration manually
Non-preset - using GraphManager directly
Training an agent with a custom Gym environment
Advanced functionality - proprietary exploration policy, checkpoint evaluation
Using Coach from the Command Line
When running Coach from the command line, we use a Preset module to define the experiment parameters.
As its name implies, a preset is a predefined set of parameters to run some agent on some environment.
Coach has many predefined presets that follow the algorithms definitions in the published papers, and allows training some of the existing algorithms with essentially no coding at all. This presets can easily be run from the command line. For example
Step1:  Preset based - using CoachInterface
The basic method to run Coach directly from python is  through a CoachInterface object, which uses the same arguments as the command line invocation but allowes for more flexibility and additional control of the training/inference process.
Let's start with some examples.
Training a preset
In this example, we'll create a very simple graph containing a Clipped PPO agent running with the CartPole-v0 Gym environment. CoachInterface has a few useful parameters such as custom_parameter that enables overriding preset settings, and other optional parameters enabling control over the training process. We'll override the preset's schedule parameters, train with a single rollout worker, and save checkpoints every 10 seconds
Step2:  Running each training or inference iteration manually
The graph manager (which was instantiated in the preset) can be accessed from the CoachInterface object. The graph manager simplifies the scheduling process by encapsulating the calls to each of the training phases. Sometimes, it can be beneficial to have a more fine grained control over the scheduling process. This can be easily done by calling the individual phase functions directly
Step3:  Sometimes we may want to track the agent's decisions, log or maybe even modify them.
We can access the agent itself through the CoachInterface as follows. 
Note that we also need an instance of the environment to do so. In this case we use instantiate a GymEnvironment object with the CartPole GymVectorEnvironment
Step4:  Non-preset - using GraphManager directly
It is also possible to invoke coach directly in the python code without defining a preset (which is necessary for CoachInterface) by using the GraphManager object directly. Using Coach this way won't allow you access functionalities such as multi-threading, but it might be convenient if you don't want to define a preset file.
Training an agent with a custom Gym environment
Here we show an example of how to use the GraphManager to train an agent on a custom Gym environment.
We first construct a GymEnvironmentParameters object describing the environment parameters. For Gym environments with vector observations, we can use the more specific GymVectorEnvironment object. 
The path to the custom environment is defined in the level parameter and it can be the absolute path to its class (e.g. '/home/user/my_environment_dir/my_environment_module.py
Step5:  Advanced functionality - proprietary exploration policy, checkpoint evaluation
Agent modules, such as exploration policy, memory and neural network topology can be replaced with proprietary ones. In this example we'll show how to replace the default exploration policy of the DQN agent with a different one that is defined under the Resources folder. We'll also show how to change the default checkpoint save settings, and how to load a checkpoint for evaluation.
We'll start with the standard definitions of a DQN agent solving the CartPole environment (taken from the Cartpole_DQN preset)
Step6:  Next, we'll override the exploration policy with our own policy defined in Resources/exploration.py.
We'll also define the checkpoint save directory and interval in seconds.
Make sure the first cell at the top of this notebook is run before the following one, such that module_path and resources_path are adding to sys path.
Step7:  Last, we'll load the latest checkpoint from the checkpoint directory, and evaluate it.
<ASSISTANT_TASK:>
Python Code:
# Adding module path to sys path if not there, so rl_coach submodules can be imported
import os
import sys
import tensorflow as tf
module_path = os.path.abspath(os.path.join('..'))
resources_path = os.path.abspath(os.path.join('Resources'))
if module_path not in sys.path:
    sys.path.append(module_path)
if resources_path not in sys.path:
    sys.path.append(resources_path)
    
from rl_coach.coach import CoachInterface
Explanation: Getting Started Guide
Table of Contents
Using Coach from the Command Line
Using Coach as a Library
Preset based - using CoachInterface
Training a preset
Running each training or inference iteration manually
Non-preset - using GraphManager directly
Training an agent with a custom Gym environment
Advanced functionality - proprietary exploration policy, checkpoint evaluation
Using Coach from the Command Line
When running Coach from the command line, we use a Preset module to define the experiment parameters.
As its name implies, a preset is a predefined set of parameters to run some agent on some environment.
Coach has many predefined presets that follow the algorithms definitions in the published papers, and allows training some of the existing algorithms with essentially no coding at all. This presets can easily be run from the command line. For example:
coach -p CartPole_DQN
You can find all the predefined presets under the presets directory, or by listing them using the following command:
coach -l
Coach can also be used with an externally defined preset by passing the absolute path to the module and the name of the graph manager object which is defined in the preset: 
coach -p /home/my_user/my_agent_dir/my_preset.py:graph_manager
Some presets are generic for multiple environment levels, and therefore require defining the specific level through the command line:
coach -p Atari_DQN -lvl breakout
There are plenty of other command line arguments you can use in order to customize the experiment. A full documentation of the available arguments can be found using the following command:
coach -h
Using Coach as a Library
Alternatively, Coach can be used a library directly from python. As described above, Coach uses the presets mechanism to define the experiments. A preset is essentially a python module which instantiates a GraphManager object. The graph manager is a container that holds the agents and the environments, and has some additional parameters for running the experiment, such as visualization parameters. The graph manager acts as the scheduler which orchestrates the experiment.
Note: Each one of the examples in this section is independent, so notebook kernels need to be restarted before running it. Make sure you run the next cell before running any of the examples.
End of explanation
coach = CoachInterface(preset='CartPole_ClippedPPO',
                       # The optional custom_parameter enables overriding preset settings
                       custom_parameter='heatup_steps=EnvironmentSteps(5);improve_steps=TrainingSteps(3)',
                       # Other optional parameters enable easy access to advanced functionalities
                       num_workers=1, checkpoint_save_secs=10)
coach.run()
Explanation: Preset based - using CoachInterface
The basic method to run Coach directly from python is  through a CoachInterface object, which uses the same arguments as the command line invocation but allowes for more flexibility and additional control of the training/inference process.
Let's start with some examples.
Training a preset
In this example, we'll create a very simple graph containing a Clipped PPO agent running with the CartPole-v0 Gym environment. CoachInterface has a few useful parameters such as custom_parameter that enables overriding preset settings, and other optional parameters enabling control over the training process. We'll override the preset's schedule parameters, train with a single rollout worker, and save checkpoints every 10 seconds:
End of explanation
from rl_coach.environments.gym_environment import GymEnvironment, GymVectorEnvironment
from rl_coach.base_parameters import VisualizationParameters
from rl_coach.core_types import EnvironmentSteps
tf.reset_default_graph()
coach = CoachInterface(preset='CartPole_ClippedPPO')
# registering an iteration signal before starting to run
coach.graph_manager.log_signal('iteration', -1)
coach.graph_manager.heatup(EnvironmentSteps(100))
# training
for it in range(10):
    # logging the iteration signal during training
    coach.graph_manager.log_signal('iteration', it)
    # using the graph manager to train and act a given number of steps
    coach.graph_manager.train_and_act(EnvironmentSteps(100))
    # reading signals during training
    training_reward = coach.graph_manager.get_signal_value('Training Reward')
Explanation: Running each training or inference iteration manually
The graph manager (which was instantiated in the preset) can be accessed from the CoachInterface object. The graph manager simplifies the scheduling process by encapsulating the calls to each of the training phases. Sometimes, it can be beneficial to have a more fine grained control over the scheduling process. This can be easily done by calling the individual phase functions directly:
End of explanation
# inference
env_params = GymVectorEnvironment(level='CartPole-v0')
env = GymEnvironment(**env_params.__dict__, visualization_parameters=VisualizationParameters())
response = env.reset_internal_state()
for _ in range(10):
    action_info = coach.graph_manager.get_agent().choose_action(response.next_state)
    print("State:{}, Action:{}".format(response.next_state,action_info.action))
    response = env.step(action_info.action)
    print("Reward:{}".format(response.reward))
Explanation: Sometimes we may want to track the agent's decisions, log or maybe even modify them.
We can access the agent itself through the CoachInterface as follows. 
Note that we also need an instance of the environment to do so. In this case we use instantiate a GymEnvironment object with the CartPole GymVectorEnvironment:
End of explanation
from rl_coach.agents.clipped_ppo_agent import ClippedPPOAgentParameters
from rl_coach.environments.gym_environment import GymVectorEnvironment
from rl_coach.graph_managers.basic_rl_graph_manager import BasicRLGraphManager
from rl_coach.graph_managers.graph_manager import SimpleSchedule
from rl_coach.architectures.embedder_parameters import InputEmbedderParameters
# Resetting tensorflow graph as the network has changed.
tf.reset_default_graph()
# define the environment parameters
bit_length = 10
env_params = GymVectorEnvironment(level='rl_coach.environments.toy_problems.bit_flip:BitFlip')
env_params.additional_simulator_parameters = {'bit_length': bit_length, 'mean_zero': True}
# Clipped PPO
agent_params = ClippedPPOAgentParameters()
agent_params.network_wrappers['main'].input_embedders_parameters = {
    'state': InputEmbedderParameters(scheme=[]),
    'desired_goal': InputEmbedderParameters(scheme=[])
}
graph_manager = BasicRLGraphManager(
    agent_params=agent_params,
    env_params=env_params,
    schedule_params=SimpleSchedule()
)
graph_manager.improve()
Explanation: Non-preset - using GraphManager directly
It is also possible to invoke coach directly in the python code without defining a preset (which is necessary for CoachInterface) by using the GraphManager object directly. Using Coach this way won't allow you access functionalities such as multi-threading, but it might be convenient if you don't want to define a preset file.
Training an agent with a custom Gym environment
Here we show an example of how to use the GraphManager to train an agent on a custom Gym environment.
We first construct a GymEnvironmentParameters object describing the environment parameters. For Gym environments with vector observations, we can use the more specific GymVectorEnvironment object. 
The path to the custom environment is defined in the level parameter and it can be the absolute path to its class (e.g. '/home/user/my_environment_dir/my_environment_module.py:MyEnvironmentClass') or the relative path to the module as in this example. In any case, we can use the custom gym environment without registering it.
Custom parameters for the environment's __init__ function can be passed as additional_simulator_parameters.
End of explanation
from rl_coach.agents.dqn_agent import DQNAgentParameters
from rl_coach.base_parameters import VisualizationParameters, TaskParameters
from rl_coach.core_types import TrainingSteps, EnvironmentEpisodes, EnvironmentSteps
from rl_coach.environments.gym_environment import GymVectorEnvironment
from rl_coach.graph_managers.basic_rl_graph_manager import BasicRLGraphManager
from rl_coach.graph_managers.graph_manager import ScheduleParameters
from rl_coach.memories.memory import MemoryGranularity
####################
# Graph Scheduling #
####################
# Resetting tensorflow graph as the network has changed.
tf.reset_default_graph()
schedule_params = ScheduleParameters()
schedule_params.improve_steps = TrainingSteps(4000)
schedule_params.steps_between_evaluation_periods = EnvironmentEpisodes(10)
schedule_params.evaluation_steps = EnvironmentEpisodes(1)
schedule_params.heatup_steps = EnvironmentSteps(1000)
#########
# Agent #
#########
agent_params = DQNAgentParameters()
# DQN params
agent_params.algorithm.num_steps_between_copying_online_weights_to_target = EnvironmentSteps(100)
agent_params.algorithm.discount = 0.99
agent_params.algorithm.num_consecutive_playing_steps = EnvironmentSteps(1)
# NN configuration
agent_params.network_wrappers['main'].learning_rate = 0.00025
agent_params.network_wrappers['main'].replace_mse_with_huber_loss = False
# ER size
agent_params.memory.max_size = (MemoryGranularity.Transitions, 40000)
################
#  Environment #
################
env_params = GymVectorEnvironment(level='CartPole-v0')
Explanation: Advanced functionality - proprietary exploration policy, checkpoint evaluation
Agent modules, such as exploration policy, memory and neural network topology can be replaced with proprietary ones. In this example we'll show how to replace the default exploration policy of the DQN agent with a different one that is defined under the Resources folder. We'll also show how to change the default checkpoint save settings, and how to load a checkpoint for evaluation.
We'll start with the standard definitions of a DQN agent solving the CartPole environment (taken from the Cartpole_DQN preset)
End of explanation
from exploration import MyExplorationParameters
# Overriding the default DQN Agent exploration policy with my exploration policy
agent_params.exploration = MyExplorationParameters()
# Creating a graph manager to train a DQN agent to solve CartPole
graph_manager = BasicRLGraphManager(agent_params=agent_params, env_params=env_params,
                                    schedule_params=schedule_params, vis_params=VisualizationParameters())
# Resources path was defined at the top of this notebook
my_checkpoint_dir = resources_path + '/checkpoints'
# Checkpoints will be stored every 5 seconds to the given directory
task_parameters1 = TaskParameters()
task_parameters1.checkpoint_save_dir = my_checkpoint_dir
task_parameters1.checkpoint_save_secs = 5
graph_manager.create_graph(task_parameters1)
graph_manager.improve()
Explanation: Next, we'll override the exploration policy with our own policy defined in Resources/exploration.py.
We'll also define the checkpoint save directory and interval in seconds.
Make sure the first cell at the top of this notebook is run before the following one, such that module_path and resources_path are adding to sys path.
End of explanation
import tensorflow as tf
import shutil
# Clearing the previous graph before creating the new one to avoid name conflicts
tf.reset_default_graph()
# Updating the graph manager's task parameters to restore the latest stored checkpoint from the checkpoints directory
task_parameters2 = TaskParameters()
task_parameters2.checkpoint_restore_path = my_checkpoint_dir
graph_manager.create_graph(task_parameters2)
graph_manager.evaluate(EnvironmentSteps(5))
# Clearning up
shutil.rmtree(my_checkpoint_dir)
Explanation: Last, we'll load the latest checkpoint from the checkpoint directory, and evaluate it.
End of explanation
<END_TASK>