<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
Step2:  Analysis, Goals, and Predictions
Here, I'd like to go over the loss functions being used to train the models and what we are aiming for, so that we can better understand the models' performance as training progresses. My approach will follow that of Ng (2015), as outlined in the textbook "Deep Learning" by Goodfellow et al.
Step3:  Single Plots Distinguishing One Variable
Step4:  Plots with Fixed Learning Rate
Step5:  Hyperparam Search on Dropout and Num Layers
Step6:  Loss Functions
I've been scratching my head a lot lately thinking about how to implement sampled softmax loss without having to split tensors across time, an operation that destroys any gain in runtime efficiency offered by negative sampling. Let's start by working through how tensorflow samples classes by default
<ASSISTANT_TASK:>
Python Code:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = 10, 8
BASE = '/home/brandon/Documents/seq2seq_projects/data/saved_train_data/'
path = BASE + 'cornell_03_11.csv'
df = pd.read_csv(path, index_col=0)
df.head()
embed_sizes = set(df['embed_size'])
state_sizes = set(df['state_size'])
learning_rates = set(df['learning_rate'])
print(embed_sizes)
print(state_sizes)
print(learning_rates)
def get_split(df, col, vals):
    return [(v, df[df[col]==v]) for v in vals]
def split_df_and_plot(df, split_col, split_vals):
    
    Example usage:
        split_df_and_plot(df, 'learning_rate', learning_rates)
    
    df_split = get_split(df, split_col, split_vals)
    plt.figure(figsize=(8, 6))
    for val, df_sp in df_split:
        ax=plt.subplot()
        plt.scatter(df_sp['global_step'], df_sp['loss'], label='%.3f' % val)
    plt.title(split_col + ' Comparisons', fontsize=20)
    ax.set_xlabel('Global Step', fontsize=15)
    ax.set_ylabel('Validation Loss', fontsize=15)
    leg = ax.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0., 
               title=split_col, prop={'size':15})
    plt.setp(leg.get_title(),fontsize=20)
    plt.tight_layout()
    plt.savefig(split_col+'.pdf', bbox_extra_artists=(leg,), bbox_inches='tight')
    plt.show()
    
def inception_split(df, split_col_one, split_vals_one, split_col_two, split_vals_two):
    ENHANCE
    df_split_one = get_split(df, split_col_one, split_vals_one)
    fig = plt.figure(figsize=(12, 10))
    ctr = 1
    for val_one, df_sp_one in df_split_one:
        df_split_two = get_split(df_sp_one, split_col_two, split_vals_two)
        ax=fig.add_subplot(3, 2, ctr)
        for val_two, df_sp_two in df_split_two:
            ax.scatter(df_sp_two['global_step'], df_sp_two['loss'], label=split_col_two + ': %.2f' % val_two)
            ax.set_ylim([3., 10.])
            plt.title(split_col_one + ' = %.2f' % val_one, fontsize=15)
            ax.set_xlabel('Global Step', fontsize=12)
            ax.set_ylabel('Validation Loss', fontsize=12)
            if ctr in [2, 4, 6]:
                leg = ax.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0., 
                       title=split_col_two, prop={'size':12})
                plt.setp(leg.get_title(),fontsize=15)
        ctr += 1
    plt.tight_layout()
    plt.savefig(split_col_one + "_" + split_col_two + '.pdf', bbox_extra_artists=(leg,), bbox_inches='tight')
    plt.show()
Explanation: Analysis, Goals, and Predictions
Here, I'd like to go over the loss functions being used to train the models and what we are aiming for, so that we can better understand the models' performance as training progresses. My approach will follow that of Ng (2015), as outlined in the textbook "Deep Learning" by Goodfellow et al.:
Determine your goals -- error metric(s) and target (re: desired) value(s). 
Establish a working end-to-end pipeline. 
Determine bottlenecks in performance, their sources, and whether they're due to overfitting/underfitting/software defect(s). 
Repeatedly make incremental changes such as gathering new data, adjusting hyperparams, or changing algorithms. 
Selecting Hyperparameters
Manual HyperParameter Tuning
Here are I'll just bullet the main ideas:
* The learning rate is perhaps the most important hyperparameter. The training error increases appx exponentially as the learning rate decreases below its optimal value. Above the optimal value, the training error basically shoots off to infinity (vertical wall). 
* Next, the best perfomance usually comes from a large model that is regularized well, for example, by using dropout. 
* Table showing typical hyperparameter relationships with model capacity. Remember that you can basically brute force your way to good performance by jacking up the model capacity and training set size. 
| Hyperparameter | Increases capacity when... | 
| -------------- | -------------------------- |
| Num hidden units | increased | 
| Learning rate | tuned optimally |
| Convolution kernal width | increased | 
| Implicit zero padding | increased | 
| Weight decay coefficient | decreased | 
| Dropout rate | decreased | 
Automatic HyperParameter Optimization
Grid Search: This is what I'm doing right now. User selects a small finite set of values to explore. Grid search trains a model for every joint specification of hyperparameter values in the Cartesian product of possible values. The experiment with the best validation error is chosen as the best. 
Random Search (Better): 
1. Define a marginal distribution for each hyperparameter, e.g. multinoulli for discrete hparams or uniform (log-scale) for positive real-valued hyparams. For example, if we were interested in the range $[10^{-5}, 0.1]$ for the learning rate:
$$
\begin{align}
\texttt{logLearningRate} &\sim Unif[-1, -5] \
\texttt{learningRate} &= 10^{logLearningRate}
\end{align}
$$
Debugging Strategies
Determining whether or not a machine learning model is broken is hard. Here are some debugging tips:
* Visualize the model in action: Not just the quantitative stuff. How do the filters look? How is the chatbot responding?
* Visualize the worst mistakes: For example, our chatbot models output probabilities for the word tokens, and we either sample or argmax. One way to get an idea of what sentences our model does poorly on is to choose examples where the output probability max is small. In other words, if argmax(output) is much lower than usual, that says our model is rather unsure what is the best next word (think of the limiting case where it outputs 1/numOutputs for all possible tokens!). 
* Fit a tiny dataset: Oooh, I like this one! Even small models can be guaranteed to be able to fit a sufficiently small dataset. Make sure you can write program that can train on say, a handful of input-output sentences, and produce the output given any of the inputs with near perfect accuracy. 
* Monitor histograms of activations/gradients: The preactivation can tell us if the units saturate, or how often they do. For tanh units, the average of the absolute value of the preactivations tells us how saturated the unit is. It is also useful to compare the parameter gradients with the parameters themselves. Ideally, we'd like the gradients over a minibatch to be about 1 percent of the magnitude of the parameter. 
Plotting the Hyperparameter-Search Results
End of explanation
split_df_and_plot(df, 'learning_rate', learning_rates)
split_df_and_plot(df, 'embed_size', embed_sizes)
split_df_and_plot(df, 'state_size', state_sizes)
Explanation: Single Plots Distinguishing One Variable
End of explanation
inception_split(df, "learning_rate", learning_rates, "state_size", state_sizes)
Explanation: Plots with Fixed Learning Rate
End of explanation
path = BASE + 'cornell.csv'
df = pd.read_csv(path, index_col=0)
df.head()
dropout_probs = set(df['dropout_prob'])
num_layers = set(df['num_layers'])
print(dropout_probs)
print(num_layers)
split_df_and_plot(df, 'dropout_prob', dropout_probs)
split_df_and_plot(df, 'num_layers', num_layers)
inception_split(df, "dropout_prob", dropout_probs, "num_layers", num_layers)
inception_split(df, "num_layers", num_layers, "dropout_prob", dropout_probs)
Explanation: Hyperparam Search on Dropout and Num Layers
End of explanation
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
vocab_size = 40000
x_axis = np.arange(vocab_size)
zipfian = (np.log(x_axis + 2) - np.log(x_axis + 1)) / np.log(vocab_size + 1)
plt.figure(figsize=(10, 6))
plt.semilogy(x_axis, zipfian)
plt.title('Log-Uniform Zipfian Sampling Distribution.')
plt.xlabel('Word ID (in order of decreasing freq.)')
plt.ylabel('Sampling probability')
plt.show()
batch_size = 64
num_samples = 512
samples = [np.random.choice(x_axis, p=zipfian) for _ in range(num_samples)]
plt.hist(samples)
plt.title('Histogram of %d Sampled Values' % num_samples)
plt.xlabel('Sampled Word ID')
plt.ylabel('Counts')
plt.show()
Explanation: Loss Functions
I've been scratching my head a lot lately thinking about how to implement sampled softmax loss without having to split tensors across time, an operation that destroys any gain in runtime efficiency offered by negative sampling. Let's start by working through how tensorflow samples classes by default: log-uniform (Zipfian) base distribution:
$$
\text{Prob(c)} = \dfrac{\log(c+2) - \log(c + 1)}{\log(\text{vocab size} + 1)}
$$
End of explanation
<END_TASK>