<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Try not to peek at the solutions when you go through the exercises. ;-)
First let's make sure this notebook works well in both Python 2 and Python 3
Step1:  From previous notebooks
Step2:  TensorBoard
Exercise 6
In this exercise, you will learn to use TensorBoard. It is a great visualization tool that comes with TensorFlow. It works by parsing special TensorFlow logs, called summaries, and displaying them nicely.
6.1) Starting the TensorBoard server. Open a Terminal and type the following commands.
Move to the tensorflow-safari-course directory
Step3:  6.4) Print out the name of a few operations. Notice how the names now have the scope as a prefix.
6.5) TensorBoard is capable of displaying data from multiple TensorFlow runs (for example multiple training sessions). For this, we need to place the data from each run in a different subdirectory of the tf_logs directory. We can name these subdirectories however we want, but a simple option is to name them using a timestamp. The following logdir() function returns the path of a subdirectory whose name is based on the current date and time
Step4:  Create a few different graphs and instantiate a different FileWriter for each one, using a different log directory every time (with the help of the logdir() function). Refresh TensorBoard and notice that you can browse any graph you want by selecting the appropriate run.
6.6) Now we will use TensorBoard to visualize the learning curve, that is the evolution of the cost function during training.
First add a scalar summary operation in the graph, using tf.summary.scalar("MSE", mse).
Next, update the training code to evaluate this scalar summary and write the result to the events file using the FileWriter's add_summary() method (also specifying the training step). For performance reasons, you probably want to do this only every 10 training iterations or so.
Next, train the model.
Refresh TensorBoard, and visit the Scalars tab. Select the appropriate run and visualize the learning curve. Try zooming in and out, and play around with the options, in particular the smoothing option.
Try not to peek at the solution below before you have done the exercise! 
Step5:  6.3)
Step6:  6.4)
Step7:  6.5)
Step8:  If we run logdir() twice within the same second, we will get the same directory name twice. To avoid this, let's wait a bit over 1 second here. In real life, this is quite unlikely to happen since training a model typically takes much longer than 1 second.
Step9:  6.6)
<ASSISTANT_TASK:>
Python Code:
from __future__ import absolute_import, division, print_function, unicode_literals
import tensorflow as tf
tf.__version__
Explanation: Try not to peek at the solutions when you go through the exercises. ;-)
First let's make sure this notebook works well in both Python 2 and Python 3:
End of explanation
learning_rate = 0.01
momentum = 0.8
Explanation: From previous notebooks
End of explanation
filenames = ["data/life_satisfaction.csv"]
n_epochs = 500
batch_size = 5
graph = tf.Graph()
with graph.as_default():
    reader = tf.TextLineReader(skip_header_lines=1)
    filename_queue = tf.train.string_input_producer(filenames, num_epochs=n_epochs)
    record_id, record = reader.read(filename_queue)
    record_defaults = [[''], [0.0], [0.0]]
    country, gdp_per_capita, life_satisfaction = tf.decode_csv(record, record_defaults=record_defaults)
    X_batch, y_batch = tf.train.batch([gdp_per_capita, life_satisfaction], batch_size=batch_size)
    X_batch_reshaped = tf.reshape(X_batch, [-1, 1])
    y_batch_reshaped = tf.reshape(y_batch, [-1, 1])
    X = tf.placeholder_with_default(X_batch_reshaped, shape=[None, 1], name="X")
    y = tf.placeholder_with_default(y_batch_reshaped, shape=[None, 1], name="y")
    b = tf.Variable(0.0, name="b")
    w = tf.Variable(tf.zeros([1, 1]), name="w")
    y_pred = tf.add(tf.matmul(X / 10000, w), b, name="y_pred")  # X @ w + b
    
    mse = tf.reduce_mean(tf.square(y_pred - y), name="mse")
    global_step = tf.Variable(0, trainable=False, name='global_step')
    optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)
    training_op = optimizer.minimize(mse, global_step=global_step)
        
    init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
    saver = tf.train.Saver()
Explanation: TensorBoard
Exercise 6
In this exercise, you will learn to use TensorBoard. It is a great visualization tool that comes with TensorFlow. It works by parsing special TensorFlow logs, called summaries, and displaying them nicely.
6.1) Starting the TensorBoard server. Open a Terminal and type the following commands.
Move to the tensorflow-safari-course directory:
~$ cd tensorflow-safari-course
Create the tf_logs directory that will hold the TensorFlow data that we will want TensorBoard to display:
~/tensorflow-safari-course$ mkdir tf_logs
Activate the virtual environment:
~/tensorflow-safari-course$ source env/bin/activate
Start the TensorBoard server:
(env) ~/tensorflow-safari-course$ tensorboard --logdir=tf_logs
Starting TensorBoard b'41' on port 6006
(You can navigate to http://127.0.1.1:6006 )
Now visit the URL given by TensorBoard. You should see the TensorBoard interface.
6.2) Now create a tf.summary.FileWriter, with the parameters: logdir="tf_logs/run_number_1/" and graph=graph where graph is the one we built just before this exercise. This will automatically:
* create the run_number_1 directory inside the tf_logs directory,
* create an events.out.tfevents.* file in that subdirectory that will contain the data that TensorBoard will display,
* write the graph's definition to this file.
Next, try refreshing the TensorBoard page in your browser (you may need to wait a couple minutes for it to detect the change, or else you can just restart the TensorBoard server). Visit the Graph tab: you should be able to visualize the graph.
6.3) As you can see, the graph looks really messy in TensorBoard. We need to organize it a bit. For this, name scopes come in handy. An operation can be placed inside a name scope in one of two ways:
Add the scope as a prefix to the operation's name, for example:
python
a = tf.constant(0.0, name="my_name_scope/a")
Or (generally clearer) use a tf.name_scope() block, for example:
python
with tf.name_scope("my_name_scope"):
    a = tf.constant(0.0, name="a")
Add name scopes to the following graph, then write it to TensorBoard (using a different run number for the log directory name) and see how much better it looks, and how much easier it is to explore.
End of explanation
from datetime import datetime
def logdir():
    root_logdir = "tf_logs"
    now = datetime.utcnow().strftime("%Y%m%d%H%M%S")
    return "{}/run_{}/".format(root_logdir, now)
logdir()
Explanation: 6.4) Print out the name of a few operations. Notice how the names now have the scope as a prefix.
6.5) TensorBoard is capable of displaying data from multiple TensorFlow runs (for example multiple training sessions). For this, we need to place the data from each run in a different subdirectory of the tf_logs directory. We can name these subdirectories however we want, but a simple option is to name them using a timestamp. The following logdir() function returns the path of a subdirectory whose name is based on the current date and time:
End of explanation
summary_writer = tf.summary.FileWriter("tf_logs/run_number_1_solution/", graph=graph)
Explanation: Create a few different graphs and instantiate a different FileWriter for each one, using a different log directory every time (with the help of the logdir() function). Refresh TensorBoard and notice that you can browse any graph you want by selecting the appropriate run.
6.6) Now we will use TensorBoard to visualize the learning curve, that is the evolution of the cost function during training.
First add a scalar summary operation in the graph, using tf.summary.scalar("MSE", mse).
Next, update the training code to evaluate this scalar summary and write the result to the events file using the FileWriter's add_summary() method (also specifying the training step). For performance reasons, you probably want to do this only every 10 training iterations or so.
Next, train the model.
Refresh TensorBoard, and visit the Scalars tab. Select the appropriate run and visualize the learning curve. Try zooming in and out, and play around with the options, in particular the smoothing option.
Try not to peek at the solution below before you have done the exercise! :)
Exercise 6 - Solution
6.1)
N/A
6.2)
End of explanation
filenames = ["data/life_satisfaction.csv"]
n_epochs = 500
batch_size = 5
graph = tf.Graph()
with graph.as_default():
    with tf.name_scope("reader"):
        reader = tf.TextLineReader(skip_header_lines=1)
        filename_queue = tf.train.string_input_producer(filenames, num_epochs=n_epochs)
        record_id, record = reader.read(filename_queue)
        record_defaults = [[''], [0.0], [0.0]]
        country, gdp_per_capita, life_satisfaction = tf.decode_csv(record, record_defaults=record_defaults)
        X_batch, y_batch = tf.train.batch([gdp_per_capita, life_satisfaction], batch_size=batch_size)
        X_batch_reshaped = tf.reshape(X_batch, [-1, 1])
        y_batch_reshaped = tf.reshape(y_batch, [-1, 1])
    with tf.name_scope("linear_model"):
        X = tf.placeholder_with_default(X_batch_reshaped, shape=[None, 1], name="X")
        y = tf.placeholder_with_default(y_batch_reshaped, shape=[None, 1], name="y")
        b = tf.Variable(0.0, name="b")
        w = tf.Variable(tf.zeros([1, 1]), name="w")
        y_pred = tf.add(tf.matmul(X / 10000, w), b, name="y_pred")  # X @ w + b
    
    with tf.name_scope("train"):
        mse = tf.reduce_mean(tf.square(y_pred - y), name="mse")
        global_step = tf.Variable(0, trainable=False, name='global_step')
        optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)
        training_op = optimizer.minimize(mse, global_step=global_step)
        
    init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
    saver = tf.train.Saver()
summary_writer = tf.summary.FileWriter("tf_logs/run_number_2_solution/", graph=graph)
Explanation: 6.3)
End of explanation
country.name, gdp_per_capita.name, X_batch.name, y_batch.name
X.name, y.name, b.name, w.name, y_pred.name
mse.name, global_step.name, training_op.name
Explanation: 6.4)
End of explanation
graph1 = tf.Graph()
with graph1.as_default():
    a = tf.constant(1.0)
summary_writer = tf.summary.FileWriter(logdir(), graph=graph)
graph2 = tf.Graph()
with graph2.as_default():
    a = tf.constant(1.0, name="a")
    b = tf.Variable(2.0, name="b")
    c = a * b
Explanation: 6.5)
End of explanation
import time
time.sleep(1.1)
summary_writer = tf.summary.FileWriter(logdir(), graph=graph)
time.sleep(1.1)
Explanation: If we run logdir() twice within the same second, we will get the same directory name twice. To avoid this, let's wait a bit over 1 second here. In real life, this is quite unlikely to happen since training a model typically takes much longer than 1 second.
End of explanation
filenames = ["data/life_satisfaction.csv"]
n_epochs = 500
batch_size = 5
graph = tf.Graph()
with graph.as_default():
    with tf.name_scope("reader"):
        reader = tf.TextLineReader(skip_header_lines=1)
        filename_queue = tf.train.string_input_producer(filenames, num_epochs=n_epochs)
        record_id, record = reader.read(filename_queue)
        record_defaults = [[''], [0.0], [0.0]]
        country, gdp_per_capita, life_satisfaction = tf.decode_csv(record, record_defaults=record_defaults)
        X_batch, y_batch = tf.train.batch([gdp_per_capita, life_satisfaction], batch_size=batch_size)
        X_batch_reshaped = tf.reshape(X_batch, [-1, 1])
        y_batch_reshaped = tf.reshape(y_batch, [-1, 1])
    with tf.name_scope("linear_model"):
        X = tf.placeholder_with_default(X_batch_reshaped, shape=[None, 1], name="X")
        y = tf.placeholder_with_default(y_batch_reshaped, shape=[None, 1], name="y")
        b = tf.Variable(0.0, name="b")
        w = tf.Variable(tf.zeros([1, 1]), name="w")
        y_pred = tf.add(tf.matmul(X / 10000, w), b, name="y_pred")
    
    with tf.name_scope("train"):
        mse = tf.reduce_mean(tf.square(y_pred - y), name="mse")
        mse_summary = tf.summary.scalar('MSE', mse)                     # <= ADDED
        global_step = tf.Variable(0, trainable=False, name='global_step')
        optimizer = tf.train.MomentumOptimizer(learning_rate, momentum)
        training_op = optimizer.minimize(mse, global_step=global_step)
        
    init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())
    saver = tf.train.Saver()
summary_writer = tf.summary.FileWriter(logdir(), graph)
with tf.Session(graph=graph) as sess:
    init.run()
    coord = tf.train.Coordinator()
    threads = tf.train.start_queue_runners(coord=coord)
    try:
        while not coord.should_stop():
            _, mse_summary_val, global_step_val = sess.run([training_op, mse_summary, global_step])
            if global_step_val % 10 == 0:
                summary_writer.add_summary(mse_summary_val, global_step_val)
    except tf.errors.OutOfRangeError:
        print("End of training")
    coord.request_stop()
    coord.join(threads)
    saver.save(sess, "./my_life_satisfaction_model")
Explanation: 6.6)
End of explanation
<END_TASK>