<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Text generation using tensor2tensor on Cloud ML Engine
This notebook illustrates using the <a href="https
Step1:  If the following cell does not reflect the version of tensorflow and tensor2tensor that you just installed, click "Reset Session" on the notebook so that the Python environment picks up the new packages.
Step2:  Download data
We will get some <a href="https
Step3:  Create training dataset
We are going to train a machine learning model to write poetry given a starting point. We'll give it one line, and it is going to tell us the next line.  So, naturally, we will train it on real poetry. Our feature will be a line of a poem and the label will be next line of that poem.
<p>
Our training dataset will consist of two files.  The first file will consist of the input lines of poetry and the other file will consist of the corresponding output lines, one output line per input line.
Step6:  We do not need to generate the data beforehand -- instead, we can have Tensor2Tensor create the training dataset for us. So, in the code below, I will use only data/poetry/raw.txt -- obviously, this allows us to productionize our model better.  Simply keep collecting raw data and generate the training/test data at the time of training.
Set up problem
The Problem in tensor2tensor is where you specify parameters like the size of your vocabulary and where to get the training data from.
Step7:  Generate training data
Our problem (translation) requires the creation of text sequences from the training dataset.  This is done using t2t-datagen and the Problem defined in the previous section.
(Ignore any runtime warnings about np.float64. they are harmless).
Step8:  Let's check to see the files that were output. If you see a broken pipe error, please ignore.
Step9:  Provide Cloud ML Engine access to data
Copy the data to Google Cloud Storage, and then provide access to the data. gsutil throws an error when removing an empty bucket, so you may see an error the first time this code is run.
Step10:  Train model locally on subset of data
Let's run it locally on a subset of the data to make sure it works.
Step11:  Note
Step12:  Option 1
Step13:  Option 2
Step14:  The job took about <b>25 minutes</b> for me and ended with these evaluation metrics
Step15:  The job took about <b>10 minutes</b> for me and ended with these evaluation metrics
Step16:  This job took <b>12 hours</b> for me and ended with these metrics
Step17:  Batch-predict
How will our poetry model do when faced with Rumi's spiritual couplets?
Step18:  Let's write out the odd-numbered lines. We'll compare how close our model can get to the beauty of Rumi's second lines given his first.
Step19:  <b> Note </b> if you get an error about "AttributeError
Step20:  Some of these are still phrases and not complete sentences. This indicates that we might need to train longer or better somehow. We need to diagnose the model ...
<p>
### Diagnosing training run
<p>
Let's diagnose the training run to see what we'd improve the next time around.
(Note that this package may not be present on Jupyter -- `pip install pydatalab` if necessary)
Step21:  <table>
<tr>
<td><img src="diagrams/poetry_loss.png"/></td>
<td><img src="diagrams/poetry_acc.png"/></td>
</table>
Looking at the loss curve, it is clear that we are overfitting (note that the orange training curve is well below the blue eval curve). Both loss curves and the accuracy-per-sequence curve, which is our key evaluation measure, plateaus after 40k. (The red curve is a faster way of computing the evaluation metric, and can be ignored). So, how do we improve the model? Well, we need to reduce overfitting and make sure the eval metrics keep going down as long as the loss is also going down.
<p>
What we really need to do is to get more data, but if that's not an option, we could try to reduce the NN and increase the dropout regularization. We could also do hyperparameter tuning on the dropout and network sizes.
## Hyperparameter tuning
tensor2tensor also supports hyperparameter tuning on Cloud ML Engine. Note the addition of the autotune flags.
<p>
The `transformer_poetry_range` was registered in problem.py above.
Step22:  When I ran the above job, it took about 15 hours and finished with these as the best parameters
Step23:  Take the first three line. I'm showing the first line of the couplet provided to the model, how the AI model that we trained complets it and how Rumi completes it
Step24:  Cloud ML Engine
Step25:  Kubeflow
Follow these instructions
<ASSISTANT_TASK:>
Python Code:
%%bash
pip freeze | grep tensor
# Choose a version of TensorFlow that is supported on TPUs
TFVERSION='1.13'
import os
os.environ['TFVERSION'] = TFVERSION
%%bash
pip install tensor2tensor==${TFVERSION} gutenberg 
# install from sou
#git clone https://github.com/tensorflow/tensor2tensor.git
#cd tensor2tensor
#yes | pip install --user -e .
Explanation: Text generation using tensor2tensor on Cloud ML Engine
This notebook illustrates using the <a href="https://github.com/tensorflow/tensor2tensor">tensor2tensor</a> library to do from-scratch, distributed training of a poetry model. Then, the trained model is used to complete new poems.
<br/>
Install tensor2tensor, and specify Google Cloud Platform project and bucket
Install the necessary packages. tensor2tensor will give us the Transformer model. Project Gutenberg gives us access to historical poems.
<b>p.s.</b> Note that this notebook uses Python2 because Project Gutenberg relies on BSD-DB which was deprecated in Python 3 and removed from the standard library.
tensor2tensor itself can be used on Python 3. It's just Project Gutenberg that has this issue.
End of explanation
%%bash
pip freeze | grep tensor
import os
PROJECT = 'cloud-training-demos' # REPLACE WITH YOUR PROJECT ID
BUCKET = 'cloud-training-demos-ml' # REPLACE WITH YOUR BUCKET NAME
REGION = 'us-central1' # REPLACE WITH YOUR BUCKET REGION e.g. us-central1
# this is what this notebook is demonstrating
PROBLEM= 'poetry_line_problem'
# for bash
os.environ['PROJECT'] = PROJECT
os.environ['BUCKET'] = BUCKET
os.environ['REGION'] = REGION
os.environ['PROBLEM'] = PROBLEM
#os.environ['PATH'] = os.environ['PATH'] + ':' + os.getcwd() + '/tensor2tensor/tensor2tensor/bin/'
%%bash
gcloud config set project $PROJECT
gcloud config set compute/region $REGION
Explanation: If the following cell does not reflect the version of tensorflow and tensor2tensor that you just installed, click "Reset Session" on the notebook so that the Python environment picks up the new packages.
End of explanation
%%bash
rm -rf data/poetry
mkdir -p data/poetry
from gutenberg.acquire import load_etext
from gutenberg.cleanup import strip_headers
import re
books = [
  # bookid, skip N lines
  (26715, 1000, 'Victorian songs'),
  (30235, 580, 'Baldwin collection'),
  (35402, 710, 'Swinburne collection'),
  (574, 15, 'Blake'),
  (1304, 172, 'Bulchevys collection'),
  (19221, 223, 'Palgrave-Pearse collection'),
  (15553, 522, 'Knowles collection') 
]
with open('data/poetry/raw.txt', 'w') as ofp:
  lineno = 0
  for (id_nr, toskip, title) in books:
    startline = lineno
    text = strip_headers(load_etext(id_nr)).strip()
    lines = text.split('\n')[toskip:]
    # any line that is all upper case is a title or author name
    # also don't want any lines with years (numbers)
    for line in lines:
      if (len(line) > 0 
          and line.upper() != line 
          and not re.match('.*[0-9]+.*', line)
          and len(line) < 50
         ):
        cleaned = re.sub('[^a-z\'\-]+', ' ', line.strip().lower())
        ofp.write(cleaned)
        ofp.write('\n')
        lineno = lineno + 1
      else:
        ofp.write('\n')
    print('Wrote lines {} to {} from {}'.format(startline, lineno, title))
!wc -l data/poetry/*.txt
Explanation: Download data
We will get some <a href="https://www.gutenberg.org/wiki/Poetry_(Bookshelf)">poetry anthologies</a> from Project Gutenberg.
End of explanation
with open('data/poetry/raw.txt', 'r') as rawfp,\
  open('data/poetry/input.txt', 'w') as infp,\
  open('data/poetry/output.txt', 'w') as outfp:
    
    prev_line = ''
    for curr_line in rawfp:
        curr_line = curr_line.strip()
        # poems break at empty lines, so this ensures we train only
        # on lines of the same poem
        if len(prev_line) > 0 and len(curr_line) > 0:       
            infp.write(prev_line + '\n')
            outfp.write(curr_line + '\n')
        prev_line = curr_line      
!head -5 data/poetry/*.txt
Explanation: Create training dataset
We are going to train a machine learning model to write poetry given a starting point. We'll give it one line, and it is going to tell us the next line.  So, naturally, we will train it on real poetry. Our feature will be a line of a poem and the label will be next line of that poem.
<p>
Our training dataset will consist of two files.  The first file will consist of the input lines of poetry and the other file will consist of the corresponding output lines, one output line per input line.
End of explanation
%%bash
rm -rf poetry
mkdir -p poetry/trainer
%%writefile poetry/trainer/problem.py
import os
import tensorflow as tf
from tensor2tensor.utils import registry
from tensor2tensor.models import transformer
from tensor2tensor.data_generators import problem
from tensor2tensor.data_generators import text_encoder
from tensor2tensor.data_generators import text_problems
from tensor2tensor.data_generators import generator_utils
tf.summary.FileWriterCache.clear() # ensure filewriter cache is clear for TensorBoard events file
@registry.register_problem
class PoetryLineProblem(text_problems.Text2TextProblem):
  Predict next line of poetry from the last line. From Gutenberg texts.
  @property
  def approx_vocab_size(self):
    return 2**13  # ~8k
  @property
  def is_generate_per_split(self):
    # generate_data will NOT shard the data into TRAIN and EVAL for us.
    return False
  @property
  def dataset_splits(self):
    Splits of data to produce and number of output shards for each.
    # 10% evaluation data
    return [{
        "split": problem.DatasetSplit.TRAIN,
        "shards": 90,
    }, {
        "split": problem.DatasetSplit.EVAL,
        "shards": 10,
    }]
  def generate_samples(self, data_dir, tmp_dir, dataset_split):
    with open('data/poetry/raw.txt', 'r') as rawfp:
      prev_line = ''
      for curr_line in rawfp:
        curr_line = curr_line.strip()
        # poems break at empty lines, so this ensures we train only
        # on lines of the same poem
        if len(prev_line) > 0 and len(curr_line) > 0:       
            yield {
                "inputs": prev_line,
                "targets": curr_line
            }
        prev_line = curr_line          
# Smaller than the typical translate model, and with more regularization
@registry.register_hparams
def transformer_poetry():
  hparams = transformer.transformer_base()
  hparams.num_hidden_layers = 2
  hparams.hidden_size = 128
  hparams.filter_size = 512
  hparams.num_heads = 4
  hparams.attention_dropout = 0.6
  hparams.layer_prepostprocess_dropout = 0.6
  hparams.learning_rate = 0.05
  return hparams
@registry.register_hparams
def transformer_poetry_tpu():
  hparams = transformer_poetry()
  transformer.update_hparams_for_tpu(hparams)
  return hparams
# hyperparameter tuning ranges
@registry.register_ranged_hparams
def transformer_poetry_range(rhp):
  rhp.set_float("learning_rate", 0.05, 0.25, scale=rhp.LOG_SCALE)
  rhp.set_int("num_hidden_layers", 2, 4)
  rhp.set_discrete("hidden_size", [128, 256, 512])
  rhp.set_float("attention_dropout", 0.4, 0.7)
%%writefile poetry/trainer/__init__.py
from . import problem
%%writefile poetry/setup.py
from setuptools import find_packages
from setuptools import setup
REQUIRED_PACKAGES = [
  'tensor2tensor'
]
setup(
    name='poetry',
    version='0.1',
    author = 'Google',
    author_email = 'training-feedback@cloud.google.com',
    install_requires=REQUIRED_PACKAGES,
    packages=find_packages(),
    include_package_data=True,
    description='Poetry Line Problem',
    requires=[]
)
!touch poetry/__init__.py
!find poetry
Explanation: We do not need to generate the data beforehand -- instead, we can have Tensor2Tensor create the training dataset for us. So, in the code below, I will use only data/poetry/raw.txt -- obviously, this allows us to productionize our model better.  Simply keep collecting raw data and generate the training/test data at the time of training.
Set up problem
The Problem in tensor2tensor is where you specify parameters like the size of your vocabulary and where to get the training data from.
End of explanation
%%bash
DATA_DIR=./t2t_data
TMP_DIR=$DATA_DIR/tmp
rm -rf $DATA_DIR $TMP_DIR
mkdir -p $DATA_DIR $TMP_DIR
# Generate data
t2t-datagen \
  --t2t_usr_dir=./poetry/trainer \
  --problem=$PROBLEM \
  --data_dir=$DATA_DIR \
  --tmp_dir=$TMP_DIR
Explanation: Generate training data
Our problem (translation) requires the creation of text sequences from the training dataset.  This is done using t2t-datagen and the Problem defined in the previous section.
(Ignore any runtime warnings about np.float64. they are harmless).
End of explanation
!ls t2t_data | head
Explanation: Let's check to see the files that were output. If you see a broken pipe error, please ignore.
End of explanation
%%bash
DATA_DIR=./t2t_data
gsutil -m rm -r gs://${BUCKET}/poetry/
gsutil -m cp ${DATA_DIR}/${PROBLEM}* ${DATA_DIR}/vocab* gs://${BUCKET}/poetry/data
%%bash
PROJECT_ID=$PROJECT
AUTH_TOKEN=$(gcloud auth print-access-token)
SVC_ACCOUNT=$(curl -X GET -H "Content-Type: application/json" \
    -H "Authorization: Bearer $AUTH_TOKEN" \
    https://ml.googleapis.com/v1/projects/${PROJECT_ID}:getConfig \
    | python -c "import json; import sys; response = json.load(sys.stdin); \
    print(response['serviceAccount'])")
echo "Authorizing the Cloud ML Service account $SVC_ACCOUNT to access files in $BUCKET"
gsutil -m defacl ch -u $SVC_ACCOUNT:R gs://$BUCKET
gsutil -m acl ch -u $SVC_ACCOUNT:R -r gs://$BUCKET  # error message (if bucket is empty) can be ignored
gsutil -m acl ch -u $SVC_ACCOUNT:W gs://$BUCKET
Explanation: Provide Cloud ML Engine access to data
Copy the data to Google Cloud Storage, and then provide access to the data. gsutil throws an error when removing an empty bucket, so you may see an error the first time this code is run.
End of explanation
%%bash
BASE=gs://${BUCKET}/poetry/data
OUTDIR=gs://${BUCKET}/poetry/subset
gsutil -m rm -r $OUTDIR
gsutil -m cp \
    ${BASE}/${PROBLEM}-train-0008* \
    ${BASE}/${PROBLEM}-dev-00000*  \
    ${BASE}/vocab* \
    $OUTDIR
Explanation: Train model locally on subset of data
Let's run it locally on a subset of the data to make sure it works.
End of explanation
%%bash
DATA_DIR=gs://${BUCKET}/poetry/subset
OUTDIR=./trained_model
rm -rf $OUTDIR
t2t-trainer \
  --data_dir=gs://${BUCKET}/poetry/subset \
  --t2t_usr_dir=./poetry/trainer \
  --problem=$PROBLEM \
  --model=transformer \
  --hparams_set=transformer_poetry \
  --output_dir=$OUTDIR --job-dir=$OUTDIR --train_steps=10
Explanation: Note: the following will work only if you are running Jupyter on a reasonably powerful machine. Don't be alarmed if your process is killed.
End of explanation
%%bash
LOCALGPU="--train_steps=7500 --worker_gpu=1 --hparams_set=transformer_poetry"
DATA_DIR=gs://${BUCKET}/poetry/data
OUTDIR=gs://${BUCKET}/poetry/model
rm -rf $OUTDIR
t2t-trainer \
  --data_dir=gs://${BUCKET}/poetry/subset \
  --t2t_usr_dir=./poetry/trainer \
  --problem=$PROBLEM \
  --model=transformer \
  --hparams_set=transformer_poetry \
  --output_dir=$OUTDIR ${LOCALGPU}
Explanation: Option 1: Train model locally on full dataset (use if running on Notebook Instance with a GPU)
You can train on the full dataset if you are on a Google Cloud Notebook Instance with a P100 or better GPU
End of explanation
%%bash
GPU="--train_steps=7500 --cloud_mlengine --worker_gpu=1 --hparams_set=transformer_poetry"
DATADIR=gs://${BUCKET}/poetry/data
OUTDIR=gs://${BUCKET}/poetry/model
JOBNAME=poetry_$(date -u +%y%m%d_%H%M%S)
echo $OUTDIR $REGION $JOBNAME
gsutil -m rm -rf $OUTDIR
yes Y | t2t-trainer \
  --data_dir=gs://${BUCKET}/poetry/subset \
  --t2t_usr_dir=./poetry/trainer \
  --problem=$PROBLEM \
  --model=transformer \
  --output_dir=$OUTDIR \
  ${GPU}
%%bash
## CHANGE the job name (based on output above: You will see a line such as Launched transformer_poetry_line_problem_t2t_20190322_233159)
gcloud ml-engine jobs describe transformer_poetry_line_problem_t2t_20190323_003001
Explanation: Option 2: Train on Cloud ML Engine
tensor2tensor has a convenient --cloud_mlengine option to kick off the training on the managed service.
It uses the Python API mentioned in the Cloud ML Engine docs, rather than requiring you to use gcloud to submit the job.
<p>
Note: your project needs P100 quota in the region.
<p>
The echo is because t2t-trainer asks you to confirm before submitting the job to the cloud. Ignore any error about "broken pipe".
If you see a message similar to this:
<pre>
    [... cloud_mlengine.py:392] Launched transformer_poetry_line_problem_t2t_20190323_000631. See console to track: https://console.cloud.google.com/mlengine/jobs/.
</pre>
then, this step has been successful.
End of explanation
%%bash
# use one of these
TPU="--train_steps=7500 --use_tpu=True --cloud_tpu_name=laktpu --hparams_set=transformer_poetry_tpu"
DATADIR=gs://${BUCKET}/poetry/data
OUTDIR=gs://${BUCKET}/poetry/model_tpu
JOBNAME=poetry_$(date -u +%y%m%d_%H%M%S)
echo $OUTDIR $REGION $JOBNAME
gsutil -m rm -rf $OUTDIR
echo "'Y'" | t2t-trainer \
  --data_dir=gs://${BUCKET}/poetry/subset \
  --t2t_usr_dir=./poetry/trainer \
  --problem=$PROBLEM \
  --model=transformer \
  --output_dir=$OUTDIR \
  ${TPU}
%%bash
gsutil ls gs://${BUCKET}/poetry/model_tpu
Explanation: The job took about <b>25 minutes</b> for me and ended with these evaluation metrics:
<pre>
Saving dict for global step 8000: global_step = 8000, loss = 6.03338, metrics-poetry_line_problem/accuracy = 0.138544, metrics-poetry_line_problem/accuracy_per_sequence = 0.0, metrics-poetry_line_problem/accuracy_top5 = 0.232037, metrics-poetry_line_problem/approx_bleu_score = 0.00492648, metrics-poetry_line_problem/neg_log_perplexity = -6.68994, metrics-poetry_line_problem/rouge_2_fscore = 0.00256089, metrics-poetry_line_problem/rouge_L_fscore = 0.128194
</pre>
Notice that accuracy_per_sequence is 0 -- Considering that we are asking the NN to be rather creative, that doesn't surprise me. Why am I looking at accuracy_per_sequence and not the other metrics? This is because it is more appropriate for problem we are solving; metrics like Bleu score are better for translation.
Option 3: Train on a directly-connected TPU
If you are running on a VM connected directly to a Cloud TPU, you can run t2t-trainer directly. Unfortunately, you won't see any output from Jupyter while the program is running.
Compare this command line to the one using GPU in the previous section.
End of explanation
%%bash
XXX This takes 3 hours on 4 GPUs. Remove this line if you are sure you want to do this.
DATADIR=gs://${BUCKET}/poetry/data
OUTDIR=gs://${BUCKET}/poetry/model_full2
JOBNAME=poetry_$(date -u +%y%m%d_%H%M%S)
echo $OUTDIR $REGION $JOBNAME
gsutil -m rm -rf $OUTDIR
echo "'Y'" | t2t-trainer \
  --data_dir=gs://${BUCKET}/poetry/subset \
  --t2t_usr_dir=./poetry/trainer \
  --problem=$PROBLEM \
  --model=transformer \
  --hparams_set=transformer_poetry \
  --output_dir=$OUTDIR \
  --train_steps=75000 --cloud_mlengine --worker_gpu=4
Explanation: The job took about <b>10 minutes</b> for me and ended with these evaluation metrics:
<pre>
Saving dict for global step 8000: global_step = 8000, loss = 6.03338, metrics-poetry_line_problem/accuracy = 0.138544, metrics-poetry_line_problem/accuracy_per_sequence = 0.0, metrics-poetry_line_problem/accuracy_top5 = 0.232037, metrics-poetry_line_problem/approx_bleu_score = 0.00492648, metrics-poetry_line_problem/neg_log_perplexity = -6.68994, metrics-poetry_line_problem/rouge_2_fscore = 0.00256089, metrics-poetry_line_problem/rouge_L_fscore = 0.128194
</pre>
Notice that accuracy_per_sequence is 0 -- Considering that we are asking the NN to be rather creative, that doesn't surprise me. Why am I looking at accuracy_per_sequence and not the other metrics? This is because it is more appropriate for problem we are solving; metrics like Bleu score are better for translation.
Option 4: Training longer
Let's train on 4 GPUs for 75,000 steps. Note the change in the last line of the job.
End of explanation
%%bash
gsutil ls gs://${BUCKET}/poetry/model   #_modeltpu
Explanation: This job took <b>12 hours</b> for me and ended with these metrics:
<pre>
global_step = 76000, loss = 4.99763, metrics-poetry_line_problem/accuracy = 0.219792, metrics-poetry_line_problem/accuracy_per_sequence = 0.0192308, metrics-poetry_line_problem/accuracy_top5 = 0.37618, metrics-poetry_line_problem/approx_bleu_score = 0.017955, metrics-poetry_line_problem/neg_log_perplexity = -5.38725, metrics-poetry_line_problem/rouge_2_fscore = 0.0325563, metrics-poetry_line_problem/rouge_L_fscore = 0.210618
</pre>
At least the accuracy per sequence is no longer zero. It is now 0.0192308 ... note that we are using a relatively small dataset (12K lines) and this is tiny in the world of natural language problems.
<p>
In order that you have your expectations set correctly: a high-performing translation model needs 400-million lines of input and takes 1 whole day on a TPU pod!
## Check trained model
End of explanation
%%writefile data/poetry/rumi.txt
Where did the handsome beloved go?
I wonder, where did that tall, shapely cypress tree go?
He spread his light among us like a candle.
Where did he go? So strange, where did he go without me?
All day long my heart trembles like a leaf.
All alone at midnight, where did that beloved go?
Go to the road, and ask any passing traveler — 
That soul-stirring companion, where did he go?
Go to the garden, and ask the gardener — 
That tall, shapely rose stem, where did he go?
Go to the rooftop, and ask the watchman — 
That unique sultan, where did he go?
Like a madman, I search in the meadows!
That deer in the meadows, where did he go?
My tearful eyes overflow like a river — 
That pearl in the vast sea, where did he go?
All night long, I implore both moon and Venus — 
That lovely face, like a moon, where did he go?
If he is mine, why is he with others?
Since he’s not here, to what “there” did he go?
If his heart and soul are joined with God,
And he left this realm of earth and water, where did he go?
Tell me clearly, Shams of Tabriz,
Of whom it is said, “The sun never dies” — where did he go?
Explanation: Batch-predict
How will our poetry model do when faced with Rumi's spiritual couplets?
End of explanation
%%bash
awk 'NR % 2 == 1' data/poetry/rumi.txt | tr '[:upper:]' '[:lower:]' | sed "s/[^a-z\'-\ ]//g" > data/poetry/rumi_leads.txt
head -3 data/poetry/rumi_leads.txt
%%bash
# same as the above training job ...
TOPDIR=gs://${BUCKET}
OUTDIR=${TOPDIR}/poetry/model #_tpu  # or ${TOPDIR}/poetry/model_full
DATADIR=${TOPDIR}/poetry/data
MODEL=transformer
HPARAMS=transformer_poetry #_tpu
# the file with the input lines
DECODE_FILE=data/poetry/rumi_leads.txt
BEAM_SIZE=4
ALPHA=0.6
t2t-decoder \
  --data_dir=$DATADIR \
  --problem=$PROBLEM \
  --model=$MODEL \
  --hparams_set=$HPARAMS \
  --output_dir=$OUTDIR \
  --t2t_usr_dir=./poetry/trainer \
  --decode_hparams="beam_size=$BEAM_SIZE,alpha=$ALPHA" \
  --decode_from_file=$DECODE_FILE
Explanation: Let's write out the odd-numbered lines. We'll compare how close our model can get to the beauty of Rumi's second lines given his first.
End of explanation
%%bash  
DECODE_FILE=data/poetry/rumi_leads.txt
cat ${DECODE_FILE}.*.decodes
Explanation: <b> Note </b> if you get an error about "AttributeError: 'HParams' object has no attribute 'problems'" please <b>Reset Session</b>, run the cell that defines the PROBLEM and run the above cell again.
End of explanation
from google.datalab.ml import TensorBoard
TensorBoard().start('gs://{}/poetry/model_full'.format(BUCKET))
for pid in TensorBoard.list()['pid']:
    TensorBoard().stop(pid)
    print('Stopped TensorBoard with pid {}'.format(pid))
Explanation: Some of these are still phrases and not complete sentences. This indicates that we might need to train longer or better somehow. We need to diagnose the model ...
<p>
### Diagnosing training run
<p>
Let's diagnose the training run to see what we'd improve the next time around.
(Note that this package may not be present on Jupyter -- `pip install pydatalab` if necessary)
End of explanation
%%bash
XXX This takes about 15 hours and consumes about 420 ML units.  Uncomment if you wish to proceed anyway
DATADIR=gs://${BUCKET}/poetry/data
OUTDIR=gs://${BUCKET}/poetry/model_hparam
JOBNAME=poetry_$(date -u +%y%m%d_%H%M%S)
echo $OUTDIR $REGION $JOBNAME
gsutil -m rm -rf $OUTDIR
echo "'Y'" | t2t-trainer \
  --data_dir=gs://${BUCKET}/poetry/subset \
  --t2t_usr_dir=./poetry/trainer \
  --problem=$PROBLEM \
  --model=transformer \
  --hparams_set=transformer_poetry \
  --output_dir=$OUTDIR \
  --hparams_range=transformer_poetry_range \
  --autotune_objective='metrics-poetry_line_problem/accuracy_per_sequence' \
  --autotune_maximize \
  --autotune_max_trials=4 \
  --autotune_parallel_trials=4 \
  --train_steps=7500 --cloud_mlengine --worker_gpu=4
Explanation: <table>
<tr>
<td><img src="diagrams/poetry_loss.png"/></td>
<td><img src="diagrams/poetry_acc.png"/></td>
</table>
Looking at the loss curve, it is clear that we are overfitting (note that the orange training curve is well below the blue eval curve). Both loss curves and the accuracy-per-sequence curve, which is our key evaluation measure, plateaus after 40k. (The red curve is a faster way of computing the evaluation metric, and can be ignored). So, how do we improve the model? Well, we need to reduce overfitting and make sure the eval metrics keep going down as long as the loss is also going down.
<p>
What we really need to do is to get more data, but if that's not an option, we could try to reduce the NN and increase the dropout regularization. We could also do hyperparameter tuning on the dropout and network sizes.
## Hyperparameter tuning
tensor2tensor also supports hyperparameter tuning on Cloud ML Engine. Note the addition of the autotune flags.
<p>
The `transformer_poetry_range` was registered in problem.py above.
End of explanation
%%bash
# same as the above training job ...
BEST_TRIAL=28  # CHANGE as needed.
TOPDIR=gs://${BUCKET}
OUTDIR=${TOPDIR}/poetry/model_hparam/$BEST_TRIAL
DATADIR=${TOPDIR}/poetry/data
MODEL=transformer
HPARAMS=transformer_poetry
# the file with the input lines
DECODE_FILE=data/poetry/rumi_leads.txt
BEAM_SIZE=4
ALPHA=0.6
t2t-decoder \
  --data_dir=$DATADIR \
  --problem=$PROBLEM \
  --model=$MODEL \
  --hparams_set=$HPARAMS \
  --output_dir=$OUTDIR \
  --t2t_usr_dir=./poetry/trainer \
  --decode_hparams="beam_size=$BEAM_SIZE,alpha=$ALPHA" \
  --decode_from_file=$DECODE_FILE \
  --hparams="num_hidden_layers=4,hidden_size=512"
%%bash  
DECODE_FILE=data/poetry/rumi_leads.txt
cat ${DECODE_FILE}.*.decodes
Explanation: When I ran the above job, it took about 15 hours and finished with these as the best parameters:
<pre>
{
      "trialId": "37",
      "hyperparameters": {
        "hp_num_hidden_layers": "4",
        "hp_learning_rate": "0.026711152525921437",
        "hp_hidden_size": "512",
        "hp_attention_dropout": "0.60589466163419292"
      },
      "finalMetric": {
        "trainingStep": "8000",
        "objectiveValue": 0.0276162791997
      }
</pre>
In other words, the accuracy per sequence achieved was 0.027 (as compared to 0.019 before hyperparameter tuning, so a <b>40% improvement!</b>) using 4 hidden layers, a learning rate of 0.0267, a hidden size of 512 and droput probability of 0.606. This is inspite of training for only 7500 steps instead of 75,000 steps ... we could train for 75k steps with these parameters, but I'll leave that as an exercise for you.
<p>
Instead, let's try predicting with this optimized model. Note the addition of the hp* flags in order to override the values hardcoded in the source code. (there is no need to specify learning rate and dropout because they are not used during inference). I am using 37 because I got the best result at trialId=37
End of explanation
%%bash
TOPDIR=gs://${BUCKET}
OUTDIR=${TOPDIR}/poetry/model_full2
DATADIR=${TOPDIR}/poetry/data
MODEL=transformer
HPARAMS=transformer_poetry
BEAM_SIZE=4
ALPHA=0.6
t2t-exporter \
  --model=$MODEL \
  --hparams_set=$HPARAMS \
  --problem=$PROBLEM \
  --t2t_usr_dir=./poetry/trainer \
  --decode_hparams="beam_size=$BEAM_SIZE,alpha=$ALPHA" \
  --data_dir=$DATADIR \
  --output_dir=$OUTDIR
%%bash
MODEL_LOCATION=$(gsutil ls gs://${BUCKET}/poetry/model_full2/export/Servo | tail -1)
echo $MODEL_LOCATION
saved_model_cli show --dir $MODEL_LOCATION --tag_set serve --signature_def serving_default
Explanation: Take the first three line. I'm showing the first line of the couplet provided to the model, how the AI model that we trained complets it and how Rumi completes it:
<p>
INPUT: where did the handsome beloved go <br/>
AI: where art thou worse to me than dead <br/>
RUMI: I wonder, where did that tall, shapely cypress tree go?
<p>
INPUT: he spread his light among us like a candle <br/>
AI: like the hurricane eclipse <br/>
RUMI: Where did he go? So strange, where did he go without me? <br/>
<p>
INPUT: all day long my heart trembles like a leaf <br/>
AI: and through their hollow aisles it plays <br/>
RUMI: All alone at midnight, where did that beloved go? 
<p>
Oh wow. The couplets as completed are quite decent considering that:
* We trained the model on American poetry, so feeding it Rumi is a bit out of left field.
* Rumi, of course, has a context and thread running through his lines while the AI (since it was fed only that one line) doesn't. 
<p>
"Spreading light like a hurricane eclipse" is a metaphor I won't soon forget. And it was created by a machine learning model!
## Serving poetry
How would you serve these predictions? There are two ways:
<ol>
<li> Use [Cloud ML Engine](https://cloud.google.com/ml-engine/docs/deploying-models) -- this is serverless and you don't have to manage any infrastructure.
<li> Use [Kubeflow](https://github.com/kubeflow/kubeflow/blob/master/user_guide.md) on Google Kubernetes Engine -- this uses clusters but will also work on-prem on your own Kubernetes cluster.
</ol>
<p>
In either case, you need to export the model first and have TensorFlow serving serve the model. The model, however, expects to see *encoded* (i.e. preprocessed) data. So, we'll do that in the Python Flask application (in AppEngine Flex) that serves the user interface.
End of explanation
%%writefile mlengine.json
description: Poetry service on ML Engine
autoScaling:
    minNodes: 1  # We don't want this model to autoscale down to zero
%%bash
MODEL_NAME="poetry"
MODEL_VERSION="v1"
MODEL_LOCATION=$(gsutil ls gs://${BUCKET}/poetry/model_full2/export/Servo | tail -1)
echo "Deleting and deploying $MODEL_NAME $MODEL_VERSION from $MODEL_LOCATION ... this will take a few minutes"
gcloud ml-engine versions delete ${MODEL_VERSION} --model ${MODEL_NAME}
#gcloud ml-engine models delete ${MODEL_NAME}
#gcloud ml-engine models create ${MODEL_NAME} --regions $REGION
gcloud alpha ml-engine versions create --machine-type=mls1-highcpu-4 ${MODEL_VERSION} \
       --model ${MODEL_NAME} --origin ${MODEL_LOCATION} --runtime-version=1.5 --config=mlengine.json
%%bash
gcloud components update --quiet
gcloud components install alpha --quiet
%%bash
MODEL_NAME="poetry"
MODEL_VERSION="v1"
MODEL_LOCATION=$(gsutil ls gs://${BUCKET}/poetry/model_full2/export/Servo | tail -1)
gcloud alpha ml-engine versions create --machine-type=mls1-highcpu-4 ${MODEL_VERSION} \
       --model ${MODEL_NAME} --origin ${MODEL_LOCATION} --runtime-version=1.5 --config=mlengine.json
Explanation: Cloud ML Engine
End of explanation
!cat application/app.yaml
%%bash
cd application
#gcloud app create  # if this is your first app
#gcloud app deploy --quiet --stop-previous-version app.yaml
Explanation: Kubeflow
Follow these instructions:
* On the GCP console, launch a Google Kubernetes Engine (GKE) cluster named 'poetry' with 2 nodes, each of which is a n1-standard-2 (2 vCPUs, 7.5 GB memory) VM
* On the GCP console, click on the Connect button for your cluster, and choose the CloudShell option
* In CloudShell, run: 
    git clone https://github.com/GoogleCloudPlatform/training-data-analyst`
    cd training-data-analyst/courses/machine_learning/deepdive/09_sequence
* Look at ./setup_kubeflow.sh and modify as appropriate.
AppEngine
What's deployed in Cloud ML Engine or Kubeflow is only the TensorFlow model. We still need a preprocessing service. That is done using AppEngine.  Edit application/app.yaml appropriately.
End of explanation
<END_TASK>