<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 TensorFlow Tutorial #21
Machine Translation
by Magnus Erik Hvass Pedersen
/ GitHub / Videos on YouTube
Introduction
Tutorial #20 showed how to use a Recurrent Neural Network (RNN) to do so-called sentiment analysis on texts of movie reviews. This tutorial will extend that idea to do Machine Translation of human languages by combining two RNN's.
You should be familiar with TensorFlow, Keras and the basics of Natural Language Processing, see Tutorials #01, #03-C and #20.
Flowchart
The following flowchart shows roughly how the neural network is constructed. It is split into two parts
Step1:  We need to import several things from Keras.
Step2:  This was developed using Python 3.6 (Anaconda) and package versions
Step3:  Load Data
We will use the Europarl data-set which has sentence-pairs in most European languages. The data was created by the European Union which translates a lot of their communications to the languages of the member-countries of the European Union.
http
Step4:  In this tutorial I have used the English-Danish data-set which contains about 2 million sentence-pairs. You can use another language by changing this language-code, see europarl.py for a list of available language-codes.
Step5:  In order for the decoder to know when to begin and end a sentence, we need to mark the start and end of each sentence with words that most likely don't occur in the data-set.
Step6:  You can change the directory for the data-files if you like.
Step7:  This will automatically download and extract the data-files if you don't have them already.
WARNING
Step8:  Load the texts for the source-language, here we use Danish.
Step9:  Load the texts for the destination-language, here we use English.
Step10:  We will build a model to translate from the source language (Danish) to the destination language (English). If you want to make the inverse translation you can merely exchange the source and destination data.
Example Data
The data is just a list of texts that is ordered so the source and destination texts match. I can confirm that this example is an accurate translation.
Step11:  Error in Data
The data-set contains about 2 million sentence-pairs. Some of the data is incorrect. This example appears to be French (or some other weird language I don't understand), although the Danish text is also included.
Step12:  Tokenizer
Neural Networks cannot work directly on text-data. We use a two-step process to convert text into numbers that can be used in a neural network. The first step is to convert text-words into so-called integer-tokens. The second step is to convert integer-tokens into vectors of floating-point numbers using a so-called embedding-layer. See Tutorial #20 for a more detailed explanation.
Set the maximum number of words in our vocabulary. This means that we will only use e.g. the 10000 most frequent words in the data-set. We use the same number for both the source and destination languages, but these could be different.
Step18:  We need a few more functions than provided by Keras' Tokenizer-class so we wrap it.
Step19:  Now create a tokenizer for the source-language. Note that we pad zeros at the beginning ('pre') of the sequences. We also reverse the sequences of tokens because the research literature suggests that this might improve performance, because the last words seen by the encoder match the first words produced by the decoder, so short-term dependencies are supposedly modelled more accurately.
Step20:  Now create the tokenizer for the destination language. We need a tokenizer for both the source- and destination-languages because their vocabularies are different. Note that this tokenizer does not reverse the sequences and it pads zeros at the end ('post') of the arrays.
Step21:  Define convenience variables for the padded token sequences. These are just 2-dimensional numpy arrays of integer-tokens.
Note that the sequence-lengths are different for the source and destination languages. This is because texts with the same meaning may have different numbers of words in the two languages. 
Furthermore, we have made a compromise when tokenizing the original texts in order to save a lot of memory. This means we only truncate about 5% of the texts.
Step22:  This is the integer-token used to mark the beginning of a text in the destination-language.
Step23:  This is the integer-token used to mark the end of a text in the destination-language.
Step24:  Example of Token Sequences
This is the output of the tokenizer. Note how it is padded with zeros at the beginning (pre-padding).
Step25:  We can reconstruct the original text by converting each integer-token back to its corresponding word
Step26:  This text is actually reversed, as can be seen when compared to the original text from the data-set
Step27:  This is the sequence of integer-tokens for the corresponding text in the destination-language. Note how it is padded with zeros at the end (post-padding).
Step28:  We can reconstruct the original text by converting each integer-token back to its corresponding word
Step29:  Compare this to the original text from the data-set, which is almost identical except for punctuation marks and a few words such as "dreaded millennium bug". This is because we only use a vocabulary of the 10000 most frequent words in the data-set and those 3 words were apparently not used frequently enough to be included in the vocabulary, so they are merely skipped.
Step30:  Training Data
Now that the data-set has been converted to sequences of integer-tokens that are padded and truncated and saved in numpy arrays, we can easily prepare the data for use in training the neural network.
The input to the encoder is merely the numpy array for the padded and truncated sequences of integer-tokens produced by the tokenizer
Step31:  The input and output data for the decoder is identical, except shifted one time-step. We can use the same numpy array to save memory by slicing it, which merely creates different 'views' of the same data in memory.
Step32:  For example, these token-sequences are identical except they are shifted one time-step.
Step33:  If we use the tokenizer to convert these sequences back into text, we see that they are identical except for the first word which is 'ssss' that marks the beginning of a text.
Step34:  Create the Neural Network
Create the Encoder
First we create the encoder-part of the neural network which maps a sequence of integer-tokens to a "thought vector". We will use the so-called functional API of Keras for this, where we first create the objects for all the layers of the neural network and then we connect them later, this allows for more flexibility than the so-called sequential API in Keras, which is useful when experimenting with more complicated architectures and ways of connecting the encoder and decoder.
This is the input for the encoder which takes batches of integer-token sequences. The None indicates that the sequences can have arbitrary length.
Step35:  This is the length of the vectors output by the embedding-layer, which maps integer-tokens to vectors of values roughly between -1 and 1, so that words that have similar semantic meanings are mapped to vectors that are similar. See Tutorial #20 for a more detailed explanation of this.
Step36:  This is the embedding-layer.
Step37:  This is the size of the internal states of the Gated Recurrent Units (GRU). The same size is used in both the encoder and decoder.
Step38:  This creates the 3 GRU layers that will map from a sequence of embedding-vectors to a single "thought vector" which summarizes the contents of the input-text. Note that the last GRU-layer does not return a sequence.
Step39:  This helper-function connects all the layers of the encoder.
Step40:  Note how the encoder uses the normal output from its last GRU-layer as the "thought vector". Research papers often use the internal state of the encoder's last recurrent layer as the "thought vector". But this makes the implementation more complicated and is not necessary when using the GRU. But if you were using the LSTM instead then it is necessary to use the LSTM's internal states as the "thought vector" because it actually has two internal vectors, which we would need to initialize the two internal states of the decoder's LSTM units.
We can now use this function to connect all the layers in the encoder so it can be connected to the decoder further below.
Step41:  Create the Decoder
Create the decoder-part which maps the "thought vector" to a sequence of integer-tokens.
The decoder takes two inputs. First it needs the "thought vector" produced by the encoder which summarizes the contents of the input-text.
Step42:  The decoder also needs a sequence of integer-tokens as inputs. During training we will supply this with a full sequence of integer-tokens e.g. corresponding to the text "ssss once upon a time eeee". 
During inference when we are translating new input-texts, we will start by feeding a sequence with just one integer-token for "ssss" which marks the beginning of a text, and combined with the "thought vector" from the encoder, the decoder will hopefully be able to produce the correct next word e.g. "once".
Step43:  This is the embedding-layer which converts integer-tokens to vectors of real-valued numbers roughly between -1 and 1. Note that we have different embedding-layers for the encoder and decoder because we have two different vocabularies and two different tokenizers for the source and destination languages.
Step44:  This creates the 3 GRU layers of the decoder. Note that they all return sequences because we ultimately want to output a sequence of integer-tokens that can be converted into a text-sequence.
Step45:  The GRU layers output a tensor with shape [batch_size, sequence_length, state_size], where each "word" is encoded as a vector of length state_size. We need to convert this into sequences of integer-tokens that can be interpreted as words from our vocabulary.
One way of doing this is to convert the GRU output to a one-hot encoded array. It works but it is extremely wasteful, because for a vocabulary of e.g. 10000 words we need a vector with 10000 elements, so we can select the index of the highest element to be the integer-token.
Step46:  The decoder is built using the functional API of Keras, which allows more flexibility in connecting the layers e.g. to route different inputs to the decoder. This is useful because we have to connect the decoder directly to the encoder, but we will also connect the decoder to another input so we can run it separately.
This function connects all the layers of the decoder to some input of the initial-state values for the GRU layers.
Step47:  Connect and Create the Models
We can now connect the encoder and decoder in different ways.
First we connect the encoder directly to the decoder so it is one whole model that can be trained end-to-end. This means the initial-state of the decoder's GRU units are set to the output of the encoder.
Step48:  Then we create a model for just the encoder alone. This is useful for mapping a sequence of integer-tokens to a "thought-vector" summarizing its contents.
Step49:  Then we create a model for just the decoder alone. This allows us to directly input the initial state for the decoder's GRU units.
Step50:  Note that all these models use the same weights and variables of the encoder and decoder. We are merely changing how they are connected. So once the entire model has been trained, we can run the encoder and decoder models separately with the trained weights.
Compile the Model
The output of the decoder is a sequence of one-hot encoded arrays. In order to train the decoder we need to supply the one-hot encoded arrays that we desire to see on the decoder's output, and then use a loss-function like cross-entropy to train the decoder to produce this desired output.
However, our data-set contains integer-tokens instead of one-hot encoded arrays. Each one-hot encoded array has 10000 elements so it would be extremely wasteful to convert the entire data-set to one-hot encoded arrays.
A better way is to use a so-called sparse cross-entropy loss-function, which does the conversion internally from integers to one-hot encoded arrays.
We have used the Adam optimizer in many of the previous tutorials, but it seems to diverge in some of these experiments with Recurrent Neural Networks. RMSprop seems to work much better for these.
Step51:  Callback Functions
During training we want to save checkpoints and log the progress to TensorBoard so we create the appropriate callbacks for Keras.
This is the callback for writing checkpoints during training.
Step52:  This is the callback for stopping the optimization when performance worsens on the validation-set.
Step53:  This is the callback for writing the TensorBoard log during training.
Step54:  Load Checkpoint
You can reload the last saved checkpoint so you don't have to train the model every time you want to use it.
Step55:  Train the Model
We wrap the data in named dicts so we are sure the data is assigned correctly to the inputs and outputs of the model.
Step56:  We want a validation-set of 10000 sequences but Keras needs this number as a fraction.
Step57:  Now we can train the model. One epoch of training took about 1 hour on a GTX 1070 GPU. You probably need to run 10 epochs or more during training. After 10 epochs the loss was about 1.10 on the training-set and about 1.15 on the validation-set.
The batch-size was chosen to keep the GPU running at nearly 100% while being within the memory limits of 8GB for this GPU.
Step59:  Translate Texts
This function translates a text from the source-language to the destination-language and optionally prints a true translation.
Step60:  Examples
Translate a text from the training-data. This translation is quite good. Note how it is not identical to the translation from the training-data, but the actual meaning is similar.
Step61:  Here is another example which is also a reasonable translation, although it has incorrectly translated the natural disasters. Note "countries of the European Union" has instead been translated as "member states" which are synonyms in this context.
Step62:  In this example we join two texts from the training-set. The model first sends this combined text through the encoder, which produces a "thought-vector" that seems to summarize both texts reasonably well so the decoder can produce a reasonable translation.
Step63:  If we reverse the order of these two texts then the meaning is not quite so clear for the latter text.
Step64:  This is an example I made up. It is a quite broken translation.
Step65:  This is another example I made up. This is a better translation even though it is perhaps a more complicated text.
Step66:  This is a text from a Danish song. It doesn't even make much sense in Danish. However the translation is probably so broken because several of the words are not in the vocabulary.
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
import math
import os
Explanation: TensorFlow Tutorial #21
Machine Translation
by Magnus Erik Hvass Pedersen
/ GitHub / Videos on YouTube
Introduction
Tutorial #20 showed how to use a Recurrent Neural Network (RNN) to do so-called sentiment analysis on texts of movie reviews. This tutorial will extend that idea to do Machine Translation of human languages by combining two RNN's.
You should be familiar with TensorFlow, Keras and the basics of Natural Language Processing, see Tutorials #01, #03-C and #20.
Flowchart
The following flowchart shows roughly how the neural network is constructed. It is split into two parts: An encoder which maps the source-text to a "thought vector" that summarizes the text's contents, which is then input to the second part of the neural network that decodes the "thought vector" to the destination-text.
The neural network cannot work directly on text so first we need to convert each word to an integer-token using a tokenizer. But the neural network cannot work on integers either, so we use a so-called Embedding Layer to convert each integer-token to a vector of floating-point values. The embedding is trained alongside the rest of the neural network to map words with similar semantic meaning to similar vectors of floating-point values.
For example, consider the Danish text "der var engang" which is the beginning of any fairytale and literally means "there was once" but is commonly translated into English as "once upon a time". We first convert the entire data-set to integer-tokens so the text "der var engang" becomes [12, 54, 1097]. Each of these integer-tokens is then mapped to an embedding-vector with e.g. 128 elements, so the integer-token 12 could for example become [0.12, -0.56, ..., 1.19] and the integer-token 54 could for example become [0.39, 0.09, ..., -0.12]. These embedding-vectors can then be input to the Recurrent Neural Network, which has 3 GRU-layers. See Tutorial #20 for a more detailed explanation.
The last GRU-layer outputs a single vector - the "thought vector" that summarizes the contents of the source-text - which is then used as the initial state of the GRU-units in the decoder-part.
The destination-text "once upon a time" is padded with special markers "ssss" and "eeee" to indicate its beginning and end, so the sequence of integer-tokens becomes [2, 337, 640, 9, 79, 3]. During training, the decoder will be given this entire sequence as input and  the desired output sequence is [337, 640, 9, 79, 3] which is the same sequence but time-shifted one step. We are trying to teach the decoder to map the "thought vector" and the start-token "ssss" (integer 2) to the next word "once" (integer 337), and then map the word "once" to the word "upon" (integer 640), and so forth.
This flow-chart depicts the main idea but does not show all the necessary details e.g. regarding the loss function which is also somewhat complicated.
Imports
End of explanation
# from tf.keras.models import Model  # This does not work!
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense, GRU, Embedding
from tensorflow.keras.optimizers import RMSprop
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, TensorBoard
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
Explanation: We need to import several things from Keras.
End of explanation
tf.__version__
tf.keras.__version__
Explanation: This was developed using Python 3.6 (Anaconda) and package versions:
End of explanation
import europarl
Explanation: Load Data
We will use the Europarl data-set which has sentence-pairs in most European languages. The data was created by the European Union which translates a lot of their communications to the languages of the member-countries of the European Union.
http://www.statmt.org/europarl/
End of explanation
language_code='da'
Explanation: In this tutorial I have used the English-Danish data-set which contains about 2 million sentence-pairs. You can use another language by changing this language-code, see europarl.py for a list of available language-codes.
End of explanation
mark_start = 'ssss '
mark_end = ' eeee'
Explanation: In order for the decoder to know when to begin and end a sentence, we need to mark the start and end of each sentence with words that most likely don't occur in the data-set.
End of explanation
# data_dir = "data/europarl/"
Explanation: You can change the directory for the data-files if you like.
End of explanation
europarl.maybe_download_and_extract(language_code=language_code)
Explanation: This will automatically download and extract the data-files if you don't have them already.
WARNING: The file for the English-Danish data-set is about 587 MB!
End of explanation
data_src = europarl.load_data(english=False,
                              language_code=language_code)
Explanation: Load the texts for the source-language, here we use Danish.
End of explanation
data_dest = europarl.load_data(english=True,
                               language_code=language_code,
                               start=mark_start,
                               end=mark_end)
Explanation: Load the texts for the destination-language, here we use English.
End of explanation
idx = 2
data_src[idx]
data_dest[idx]
Explanation: We will build a model to translate from the source language (Danish) to the destination language (English). If you want to make the inverse translation you can merely exchange the source and destination data.
Example Data
The data is just a list of texts that is ordered so the source and destination texts match. I can confirm that this example is an accurate translation.
End of explanation
idx = 8002
data_src[idx]
data_dest[idx]
Explanation: Error in Data
The data-set contains about 2 million sentence-pairs. Some of the data is incorrect. This example appears to be French (or some other weird language I don't understand), although the Danish text is also included.
End of explanation
num_words = 10000
Explanation: Tokenizer
Neural Networks cannot work directly on text-data. We use a two-step process to convert text into numbers that can be used in a neural network. The first step is to convert text-words into so-called integer-tokens. The second step is to convert integer-tokens into vectors of floating-point numbers using a so-called embedding-layer. See Tutorial #20 for a more detailed explanation.
Set the maximum number of words in our vocabulary. This means that we will only use e.g. the 10000 most frequent words in the data-set. We use the same number for both the source and destination languages, but these could be different.
End of explanation
class TokenizerWrap(Tokenizer):
    Wrap the Tokenizer-class from Keras with more functionality.
    
    def __init__(self, texts, padding,
                 reverse=False, num_words=None):
        
        :param texts: List of strings. This is the data-set.
        :param padding: Either 'post' or 'pre' padding.
        :param reverse: Boolean whether to reverse token-lists.
        :param num_words: Max number of words to use.
        
        Tokenizer.__init__(self, num_words=num_words)
        # Create the vocabulary from the texts.
        self.fit_on_texts(texts)
        # Create inverse lookup from integer-tokens to words.
        self.index_to_word = dict(zip(self.word_index.values(),
                                      self.word_index.keys()))
        # Convert all texts to lists of integer-tokens.
        # Note that the sequences may have different lengths.
        self.tokens = self.texts_to_sequences(texts)
        if reverse:
            # Reverse the token-sequences.
            self.tokens = [list(reversed(x)) for x in self.tokens]
        
            # Sequences that are too long should now be truncated
            # at the beginning, which corresponds to the end of
            # the original sequences.
            truncating = 'pre'
        else:
            # Sequences that are too long should be truncated
            # at the end.
            truncating = 'post'
        # The number of integer-tokens in each sequence.
        self.num_tokens = [len(x) for x in self.tokens]
        # Max number of tokens to use in all sequences.
        # We will pad / truncate all sequences to this length.
        # This is a compromise so we save a lot of memory and
        # only have to truncate maybe 5% of all the sequences.
        self.max_tokens = np.mean(self.num_tokens) \
                          + 2 * np.std(self.num_tokens)
        self.max_tokens = int(self.max_tokens)
        # Pad / truncate all token-sequences to the given length.
        # This creates a 2-dim numpy matrix that is easier to use.
        self.tokens_padded = pad_sequences(self.tokens,
                                           maxlen=self.max_tokens,
                                           padding=padding,
                                           truncating=truncating)
    def token_to_word(self, token):
        Lookup a single word from an integer-token.
        word = " " if token == 0 else self.index_to_word[token]
        return word 
    def tokens_to_string(self, tokens):
        Convert a list of integer-tokens to a string.
        # Create a list of the individual words.
        words = [self.index_to_word[token]
                 for token in tokens
                 if token != 0]
        
        # Concatenate the words to a single string
        # with space between all the words.
        text = " ".join(words)
        return text
    
    def text_to_tokens(self, text, reverse=False, padding=False):
        
        Convert a single text-string to tokens with optional
        reversal and padding.
        
        # Convert to tokens. Note that we assume there is only
        # a single text-string so we wrap it in a list.
        tokens = self.texts_to_sequences([text])
        tokens = np.array(tokens)
        if reverse:
            # Reverse the tokens.
            tokens = np.flip(tokens, axis=1)
            # Sequences that are too long should now be truncated
            # at the beginning, which corresponds to the end of
            # the original sequences.
            truncating = 'pre'
        else:
            # Sequences that are too long should be truncated
            # at the end.
            truncating = 'post'
        if padding:
            # Pad and truncate sequences to the given length.
            tokens = pad_sequences(tokens,
                                   maxlen=self.max_tokens,
                                   padding='pre',
                                   truncating=truncating)
        return tokens
Explanation: We need a few more functions than provided by Keras' Tokenizer-class so we wrap it.
End of explanation
%%time
tokenizer_src = TokenizerWrap(texts=data_src,
                              padding='pre',
                              reverse=True,
                              num_words=num_words)
Explanation: Now create a tokenizer for the source-language. Note that we pad zeros at the beginning ('pre') of the sequences. We also reverse the sequences of tokens because the research literature suggests that this might improve performance, because the last words seen by the encoder match the first words produced by the decoder, so short-term dependencies are supposedly modelled more accurately.
End of explanation
%%time
tokenizer_dest = TokenizerWrap(texts=data_dest,
                               padding='post',
                               reverse=False,
                               num_words=num_words)
Explanation: Now create the tokenizer for the destination language. We need a tokenizer for both the source- and destination-languages because their vocabularies are different. Note that this tokenizer does not reverse the sequences and it pads zeros at the end ('post') of the arrays.
End of explanation
tokens_src = tokenizer_src.tokens_padded
tokens_dest = tokenizer_dest.tokens_padded
print(tokens_src.shape)
print(tokens_dest.shape)
Explanation: Define convenience variables for the padded token sequences. These are just 2-dimensional numpy arrays of integer-tokens.
Note that the sequence-lengths are different for the source and destination languages. This is because texts with the same meaning may have different numbers of words in the two languages. 
Furthermore, we have made a compromise when tokenizing the original texts in order to save a lot of memory. This means we only truncate about 5% of the texts.
End of explanation
token_start = tokenizer_dest.word_index[mark_start.strip()]
token_start
Explanation: This is the integer-token used to mark the beginning of a text in the destination-language.
End of explanation
token_end = tokenizer_dest.word_index[mark_end.strip()]
token_end
Explanation: This is the integer-token used to mark the end of a text in the destination-language.
End of explanation
idx = 2
tokens_src[idx]
Explanation: Example of Token Sequences
This is the output of the tokenizer. Note how it is padded with zeros at the beginning (pre-padding).
End of explanation
tokenizer_src.tokens_to_string(tokens_src[idx])
Explanation: We can reconstruct the original text by converting each integer-token back to its corresponding word:
End of explanation
data_src[idx]
Explanation: This text is actually reversed, as can be seen when compared to the original text from the data-set:
End of explanation
tokens_dest[idx]
Explanation: This is the sequence of integer-tokens for the corresponding text in the destination-language. Note how it is padded with zeros at the end (post-padding).
End of explanation
tokenizer_dest.tokens_to_string(tokens_dest[idx])
Explanation: We can reconstruct the original text by converting each integer-token back to its corresponding word:
End of explanation
data_dest[idx]
Explanation: Compare this to the original text from the data-set, which is almost identical except for punctuation marks and a few words such as "dreaded millennium bug". This is because we only use a vocabulary of the 10000 most frequent words in the data-set and those 3 words were apparently not used frequently enough to be included in the vocabulary, so they are merely skipped.
End of explanation
encoder_input_data = tokens_src
Explanation: Training Data
Now that the data-set has been converted to sequences of integer-tokens that are padded and truncated and saved in numpy arrays, we can easily prepare the data for use in training the neural network.
The input to the encoder is merely the numpy array for the padded and truncated sequences of integer-tokens produced by the tokenizer:
End of explanation
decoder_input_data = tokens_dest[:, :-1]
decoder_input_data.shape
decoder_output_data = tokens_dest[:, 1:]
decoder_output_data.shape
Explanation: The input and output data for the decoder is identical, except shifted one time-step. We can use the same numpy array to save memory by slicing it, which merely creates different 'views' of the same data in memory.
End of explanation
idx = 2
decoder_input_data[idx]
decoder_output_data[idx]
Explanation: For example, these token-sequences are identical except they are shifted one time-step.
End of explanation
tokenizer_dest.tokens_to_string(decoder_input_data[idx])
tokenizer_dest.tokens_to_string(decoder_output_data[idx])
Explanation: If we use the tokenizer to convert these sequences back into text, we see that they are identical except for the first word which is 'ssss' that marks the beginning of a text.
End of explanation
encoder_input = Input(shape=(None, ), name='encoder_input')
Explanation: Create the Neural Network
Create the Encoder
First we create the encoder-part of the neural network which maps a sequence of integer-tokens to a "thought vector". We will use the so-called functional API of Keras for this, where we first create the objects for all the layers of the neural network and then we connect them later, this allows for more flexibility than the so-called sequential API in Keras, which is useful when experimenting with more complicated architectures and ways of connecting the encoder and decoder.
This is the input for the encoder which takes batches of integer-token sequences. The None indicates that the sequences can have arbitrary length.
End of explanation
embedding_size = 128
Explanation: This is the length of the vectors output by the embedding-layer, which maps integer-tokens to vectors of values roughly between -1 and 1, so that words that have similar semantic meanings are mapped to vectors that are similar. See Tutorial #20 for a more detailed explanation of this.
End of explanation
encoder_embedding = Embedding(input_dim=num_words,
                              output_dim=embedding_size,
                              name='encoder_embedding')
Explanation: This is the embedding-layer.
End of explanation
state_size = 512
Explanation: This is the size of the internal states of the Gated Recurrent Units (GRU). The same size is used in both the encoder and decoder.
End of explanation
encoder_gru1 = GRU(state_size, name='encoder_gru1',
                   return_sequences=True)
encoder_gru2 = GRU(state_size, name='encoder_gru2',
                   return_sequences=True)
encoder_gru3 = GRU(state_size, name='encoder_gru3',
                   return_sequences=False)
Explanation: This creates the 3 GRU layers that will map from a sequence of embedding-vectors to a single "thought vector" which summarizes the contents of the input-text. Note that the last GRU-layer does not return a sequence.
End of explanation
def connect_encoder():
    # Start the neural network with its input-layer.
    net = encoder_input
    
    # Connect the embedding-layer.
    net = encoder_embedding(net)
    # Connect all the GRU-layers.
    net = encoder_gru1(net)
    net = encoder_gru2(net)
    net = encoder_gru3(net)
    # This is the output of the encoder.
    encoder_output = net
    
    return encoder_output
Explanation: This helper-function connects all the layers of the encoder.
End of explanation
encoder_output = connect_encoder()
Explanation: Note how the encoder uses the normal output from its last GRU-layer as the "thought vector". Research papers often use the internal state of the encoder's last recurrent layer as the "thought vector". But this makes the implementation more complicated and is not necessary when using the GRU. But if you were using the LSTM instead then it is necessary to use the LSTM's internal states as the "thought vector" because it actually has two internal vectors, which we would need to initialize the two internal states of the decoder's LSTM units.
We can now use this function to connect all the layers in the encoder so it can be connected to the decoder further below.
End of explanation
decoder_initial_state = Input(shape=(state_size,),
                              name='decoder_initial_state')
Explanation: Create the Decoder
Create the decoder-part which maps the "thought vector" to a sequence of integer-tokens.
The decoder takes two inputs. First it needs the "thought vector" produced by the encoder which summarizes the contents of the input-text.
End of explanation
decoder_input = Input(shape=(None, ), name='decoder_input')
Explanation: The decoder also needs a sequence of integer-tokens as inputs. During training we will supply this with a full sequence of integer-tokens e.g. corresponding to the text "ssss once upon a time eeee". 
During inference when we are translating new input-texts, we will start by feeding a sequence with just one integer-token for "ssss" which marks the beginning of a text, and combined with the "thought vector" from the encoder, the decoder will hopefully be able to produce the correct next word e.g. "once".
End of explanation
decoder_embedding = Embedding(input_dim=num_words,
                              output_dim=embedding_size,
                              name='decoder_embedding')
Explanation: This is the embedding-layer which converts integer-tokens to vectors of real-valued numbers roughly between -1 and 1. Note that we have different embedding-layers for the encoder and decoder because we have two different vocabularies and two different tokenizers for the source and destination languages.
End of explanation
decoder_gru1 = GRU(state_size, name='decoder_gru1',
                   return_sequences=True)
decoder_gru2 = GRU(state_size, name='decoder_gru2',
                   return_sequences=True)
decoder_gru3 = GRU(state_size, name='decoder_gru3',
                   return_sequences=True)
Explanation: This creates the 3 GRU layers of the decoder. Note that they all return sequences because we ultimately want to output a sequence of integer-tokens that can be converted into a text-sequence.
End of explanation
decoder_dense = Dense(num_words,
                      activation='softmax',
                      name='decoder_output')
Explanation: The GRU layers output a tensor with shape [batch_size, sequence_length, state_size], where each "word" is encoded as a vector of length state_size. We need to convert this into sequences of integer-tokens that can be interpreted as words from our vocabulary.
One way of doing this is to convert the GRU output to a one-hot encoded array. It works but it is extremely wasteful, because for a vocabulary of e.g. 10000 words we need a vector with 10000 elements, so we can select the index of the highest element to be the integer-token.
End of explanation
def connect_decoder(initial_state):
    # Start the decoder-network with its input-layer.
    net = decoder_input
    # Connect the embedding-layer.
    net = decoder_embedding(net)
    
    # Connect all the GRU-layers.
    net = decoder_gru1(net, initial_state=initial_state)
    net = decoder_gru2(net, initial_state=initial_state)
    net = decoder_gru3(net, initial_state=initial_state)
    # Connect the final dense layer that converts to
    # one-hot encoded arrays.
    decoder_output = decoder_dense(net)
    
    return decoder_output
Explanation: The decoder is built using the functional API of Keras, which allows more flexibility in connecting the layers e.g. to route different inputs to the decoder. This is useful because we have to connect the decoder directly to the encoder, but we will also connect the decoder to another input so we can run it separately.
This function connects all the layers of the decoder to some input of the initial-state values for the GRU layers.
End of explanation
decoder_output = connect_decoder(initial_state=encoder_output)
model_train = Model(inputs=[encoder_input, decoder_input],
                    outputs=[decoder_output])
Explanation: Connect and Create the Models
We can now connect the encoder and decoder in different ways.
First we connect the encoder directly to the decoder so it is one whole model that can be trained end-to-end. This means the initial-state of the decoder's GRU units are set to the output of the encoder.
End of explanation
model_encoder = Model(inputs=[encoder_input],
                      outputs=[encoder_output])
Explanation: Then we create a model for just the encoder alone. This is useful for mapping a sequence of integer-tokens to a "thought-vector" summarizing its contents.
End of explanation
decoder_output = connect_decoder(initial_state=decoder_initial_state)
model_decoder = Model(inputs=[decoder_input, decoder_initial_state],
                      outputs=[decoder_output])
Explanation: Then we create a model for just the decoder alone. This allows us to directly input the initial state for the decoder's GRU units.
End of explanation
model_train.compile(optimizer=RMSprop(lr=1e-3),
                    loss='sparse_categorical_crossentropy')
Explanation: Note that all these models use the same weights and variables of the encoder and decoder. We are merely changing how they are connected. So once the entire model has been trained, we can run the encoder and decoder models separately with the trained weights.
Compile the Model
The output of the decoder is a sequence of one-hot encoded arrays. In order to train the decoder we need to supply the one-hot encoded arrays that we desire to see on the decoder's output, and then use a loss-function like cross-entropy to train the decoder to produce this desired output.
However, our data-set contains integer-tokens instead of one-hot encoded arrays. Each one-hot encoded array has 10000 elements so it would be extremely wasteful to convert the entire data-set to one-hot encoded arrays.
A better way is to use a so-called sparse cross-entropy loss-function, which does the conversion internally from integers to one-hot encoded arrays.
We have used the Adam optimizer in many of the previous tutorials, but it seems to diverge in some of these experiments with Recurrent Neural Networks. RMSprop seems to work much better for these.
End of explanation
path_checkpoint = '21_checkpoint.keras'
callback_checkpoint = ModelCheckpoint(filepath=path_checkpoint,
                                      monitor='val_loss',
                                      verbose=1,
                                      save_weights_only=True,
                                      save_best_only=True)
Explanation: Callback Functions
During training we want to save checkpoints and log the progress to TensorBoard so we create the appropriate callbacks for Keras.
This is the callback for writing checkpoints during training.
End of explanation
callback_early_stopping = EarlyStopping(monitor='val_loss',
                                        patience=3, verbose=1)
Explanation: This is the callback for stopping the optimization when performance worsens on the validation-set.
End of explanation
callback_tensorboard = TensorBoard(log_dir='./21_logs/',
                                   histogram_freq=0,
                                   write_graph=False)
callbacks = [callback_early_stopping,
             callback_checkpoint,
             callback_tensorboard]
Explanation: This is the callback for writing the TensorBoard log during training.
End of explanation
try:
    model_train.load_weights(path_checkpoint)
except Exception as error:
    print("Error trying to load checkpoint.")
    print(error)
Explanation: Load Checkpoint
You can reload the last saved checkpoint so you don't have to train the model every time you want to use it.
End of explanation
x_data = \
{
    'encoder_input': encoder_input_data,
    'decoder_input': decoder_input_data
}
y_data = \
{
    'decoder_output': decoder_output_data
}
Explanation: Train the Model
We wrap the data in named dicts so we are sure the data is assigned correctly to the inputs and outputs of the model.
End of explanation
validation_split = 10000 / len(encoder_input_data)
validation_split
Explanation: We want a validation-set of 10000 sequences but Keras needs this number as a fraction.
End of explanation
model_train.fit(x=x_data,
                y=y_data,
                batch_size=384,
                epochs=10,
                validation_split=validation_split,
                callbacks=callbacks)
Explanation: Now we can train the model. One epoch of training took about 1 hour on a GTX 1070 GPU. You probably need to run 10 epochs or more during training. After 10 epochs the loss was about 1.10 on the training-set and about 1.15 on the validation-set.
The batch-size was chosen to keep the GPU running at nearly 100% while being within the memory limits of 8GB for this GPU.
End of explanation
def translate(input_text, true_output_text=None):
    Translate a single text-string.
    # Convert the input-text to integer-tokens.
    # Note the sequence of tokens has to be reversed.
    # Padding is probably not necessary.
    input_tokens = tokenizer_src.text_to_tokens(text=input_text,
                                                reverse=True,
                                                padding=True)
    
    # Get the output of the encoder's GRU which will be
    # used as the initial state in the decoder's GRU.
    # This could also have been the encoder's final state
    # but that is really only necessary if the encoder
    # and decoder use the LSTM instead of GRU because
    # the LSTM has two internal states.
    initial_state = model_encoder.predict(input_tokens)
    # Max number of tokens / words in the output sequence.
    max_tokens = tokenizer_dest.max_tokens
    # Pre-allocate the 2-dim array used as input to the decoder.
    # This holds just a single sequence of integer-tokens,
    # but the decoder-model expects a batch of sequences.
    shape = (1, max_tokens)
    decoder_input_data = np.zeros(shape=shape, dtype=np.int)
    # The first input-token is the special start-token for 'ssss '.
    token_int = token_start
    # Initialize an empty output-text.
    output_text = ''
    # Initialize the number of tokens we have processed.
    count_tokens = 0
    # While we haven't sampled the special end-token for ' eeee'
    # and we haven't processed the max number of tokens.
    while token_int != token_end and count_tokens < max_tokens:
        # Update the input-sequence to the decoder
        # with the last token that was sampled.
        # In the first iteration this will set the
        # first element to the start-token.
        decoder_input_data[0, count_tokens] = token_int
        # Wrap the input-data in a dict for clarity and safety,
        # so we are sure we input the data in the right order.
        x_data = \
        {
            'decoder_initial_state': initial_state,
            'decoder_input': decoder_input_data
        }
        # Note that we input the entire sequence of tokens
        # to the decoder. This wastes a lot of computation
        # because we are only interested in the last input
        # and output. We could modify the code to return
        # the GRU-states when calling predict() and then
        # feeding these GRU-states as well the next time
        # we call predict(), but it would make the code
        # much more complicated.
        # Input this data to the decoder and get the predicted output.
        decoder_output = model_decoder.predict(x_data)
        # Get the last predicted token as a one-hot encoded array.
        token_onehot = decoder_output[0, count_tokens, :]
        
        # Convert to an integer-token.
        token_int = np.argmax(token_onehot)
        # Lookup the word corresponding to this integer-token.
        sampled_word = tokenizer_dest.token_to_word(token_int)
        # Append the word to the output-text.
        output_text += " " + sampled_word
        # Increment the token-counter.
        count_tokens += 1
    # Sequence of tokens output by the decoder.
    output_tokens = decoder_input_data[0]
    
    # Print the input-text.
    print("Input text:")
    print(input_text)
    print()
    # Print the translated output-text.
    print("Translated text:")
    print(output_text)
    print()
    # Optionally print the true translated text.
    if true_output_text is not None:
        print("True output text:")
        print(true_output_text)
        print()
Explanation: Translate Texts
This function translates a text from the source-language to the destination-language and optionally prints a true translation.
End of explanation
idx = 3
translate(input_text=data_src[idx],
          true_output_text=data_dest[idx])
Explanation: Examples
Translate a text from the training-data. This translation is quite good. Note how it is not identical to the translation from the training-data, but the actual meaning is similar.
End of explanation
idx = 4
translate(input_text=data_src[idx],
          true_output_text=data_dest[idx])
Explanation: Here is another example which is also a reasonable translation, although it has incorrectly translated the natural disasters. Note "countries of the European Union" has instead been translated as "member states" which are synonyms in this context.
End of explanation
idx = 3
translate(input_text=data_src[idx] + data_src[idx+1],
          true_output_text=data_dest[idx] + data_dest[idx+1])
Explanation: In this example we join two texts from the training-set. The model first sends this combined text through the encoder, which produces a "thought-vector" that seems to summarize both texts reasonably well so the decoder can produce a reasonable translation.
End of explanation
idx = 3
translate(input_text=data_src[idx+1] + data_src[idx],
          true_output_text=data_dest[idx+1] + data_dest[idx])
Explanation: If we reverse the order of these two texts then the meaning is not quite so clear for the latter text.
End of explanation
translate(input_text="der var engang et land der hed Danmark",
          true_output_text='Once there was a country named Denmark')
Explanation: This is an example I made up. It is a quite broken translation.
End of explanation
translate(input_text="Idag kan man læse i avisen at Danmark er blevet fornuftigt",
          true_output_text="Today you can read in the newspaper that Denmark has become sensible.")
Explanation: This is another example I made up. This is a better translation even though it is perhaps a more complicated text.
End of explanation
translate(input_text="Hvem spæner ud af en butik og tygger de stærkeste bolcher?",
          true_output_text="Who runs out of a shop and chews the strongest bon-bons?")
Explanation: This is a text from a Danish song. It doesn't even make much sense in Danish. However the translation is probably so broken because several of the words are not in the vocabulary.
End of explanation
<END_TASK>