<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Text Models with Keras
Dense vector word embeddings
A dense vector word embedding means we represent words with number-full numerical vectors-- most components are nonzero.  This is in contrast to  sparse vector, or bag-of-word embeddings, which have very high-dimensional vectors (the size of the vocabulary) yet with most components zero.
Dense vector models also capture word meaning, such that similar words (car and automobile) have similar numerical vectors.  In a sparse vector representation, similar words probably have completely different numerical vectors.  Dense vectors are formed as a by-product of some prediction task.  The quality of the embedding depends on both the prediction task and the data set upon which the prediction task was trained. 
When we use word embeddings in our deep learning models, we refer to their birthplace as the embedding layer.  Sometimes, we don't actually care about the trained predictor (skip-gram and cbow models); we're just interested in the embeddings by-product for use elsewhere.  Other times, we need an embedding layer to represent words in a larger model such as a sentiment classifier; there, we may opt for pre-trained dense vectors.  
When we don't care about the trained model and just want to create meaningful, dense word vectors, there are two popular prediction models
Step1:  skip-gram
Step2:  ```python
word1 = Input(shape=(1,), dtype='int64', name='word1')
word2 = Input(shape=(1,), dtype='int64', name='word2')
shared_embedding = Embedding(
    input_dim=VOCAB_SIZE+1, 
    output_dim=DENSEVEC_DIM, 
    input_length=1, 
    embeddings_constraint = unit_norm(),
    name='shared_embedding')
embedded_w1 = shared_embedding(word1)
embedded_w2 = shared_embedding(word2)
w1 = Flatten()(embedded_w1)
w2 = Flatten()(embedded_w2)
dotted = Dot(axes=1, name='dot_product')([w1, w2])
prediction = Dense(1, activation='sigmoid', name='output_layer')(dotted)
sg_model = Model(inputs=[word1, word2], outputs=prediction)
```
fastText
```python
ft_model = Sequential()
ft_model.add(Embedding(
    input_dim = MAX_FEATURES,
    output_dim = EMBEDDING_DIMS,
    input_length= MAXLEN))
ft_model.add(GlobalAveragePooling1D())
ft_model.add(Dense(1, activation='sigmoid'))
```
Models and Training data construction
The first step for CBOW and skip-gram
Our training corpus is a collection of sentences, Tweets, emails, comments, or even longer documents.  It is something composed of words.  Each word takes is turn being the "target" word, and we collect the n words behind it and n words which follow it.  This n is referred to as window size.  If our example document is the sentence "I love deep learning" and the window size is 1, we'd get
Step3:  Note word id's are numbered from 1, not zero
Step4:  Model parameters
Step5:  Model build
Step6:  Create a dense vector for each word in the pair.  The output of Embedding has shape (batch_size, sequence_length, output_dim) which in our case is (batch_size, 1, DENSEVEC_DIM).  We'll use Flatten to get rid of that pesky middle dimension (1), so going into the dot product we'll have shape (batch_size, DENSEVEC_DIM).
Step7:  At this point you can check out how the data flows through your compiled model.
Step8:  Let's try training it with our toy data set!
Step9:  Continuous Bag of Words (CBOW) model
CBOW means we take all the words in the window and use them to predict the target word.  Note we are trying to predict an actual word (or a probability distribution over words) with CBOW, whereas in skip-gram we are trying to predict a similarity score. 
FastText Model
FastText is creating dense document vectors using the words in the document enhanced with n-grams.  These are embedded, averaged, and fed through a hidden dense layer, with a sigmoid activation.  The prediction task is some binary classification of the documents.  As usual, after training we can extract the dense vectors from the model.
FastText Model Data Prep
Step12:  Some data prep functions lifted from the example
Step13:  load canned training data
Step14:  Add n-gram features
Step15:  Assign id's to the new features
Step16:  Update MAX_FEATURES
Step17:  Add n-grams to the input data
Step18:  Make all input sequences the same length by padding with zeros
Step19:  FastText Model
Step20:  fastText classifier vs. convolutional neural network (CNN) vs. long short-term memory (LSTM) classifier
Step21:  Diagram from Convolutional Neural Networks for Sentence Classification, Kim Yoon (2014)
A CNN sentence classifier
Step22:  Canned input data
Step23:  Model build
Step24:  An LSTM sentence classifier
Step25:  Appendix
<ASSISTANT_TASK:>
Python Code:
from IPython.display import Image
Explanation: Text Models with Keras
Dense vector word embeddings
A dense vector word embedding means we represent words with number-full numerical vectors-- most components are nonzero.  This is in contrast to  sparse vector, or bag-of-word embeddings, which have very high-dimensional vectors (the size of the vocabulary) yet with most components zero.
Dense vector models also capture word meaning, such that similar words (car and automobile) have similar numerical vectors.  In a sparse vector representation, similar words probably have completely different numerical vectors.  Dense vectors are formed as a by-product of some prediction task.  The quality of the embedding depends on both the prediction task and the data set upon which the prediction task was trained. 
When we use word embeddings in our deep learning models, we refer to their birthplace as the embedding layer.  Sometimes, we don't actually care about the trained predictor (skip-gram and cbow models); we're just interested in the embeddings by-product for use elsewhere.  Other times, we need an embedding layer to represent words in a larger model such as a sentiment classifier; there, we may opt for pre-trained dense vectors.  
When we don't care about the trained model and just want to create meaningful, dense word vectors, there are two popular prediction models: skip-gram and CBOW (continuous bag of words).  Word embeddings constructed in this manner are termed word2vec or w2v.  We will also look at another more recent method, fastText.  In any case, we've first got to construct training data from our corpus.  The exact procedure depends on the model.
Keras Models
Let's have a look at the Keras models we'll use in this section.  (I'm keeping the code as markup since we haven't defined any of the parameters yet.  We'll run this code after we develop input data and parameters.)
End of explanation
   Image('diagrams/skip-gram.png')
Explanation: skip-gram
End of explanation
from keras.preprocessing.sequence import skipgrams
from keras.preprocessing.text import Tokenizer, text_to_word_sequence
text1 = "I love deep learning."
text2 = "Read Douglas Adams as much as possible."
tokenizer = Tokenizer()
tokenizer.fit_on_texts([text1, text2])
word2id = tokenizer.word_index
word2id.items()
Explanation: ```python
word1 = Input(shape=(1,), dtype='int64', name='word1')
word2 = Input(shape=(1,), dtype='int64', name='word2')
shared_embedding = Embedding(
    input_dim=VOCAB_SIZE+1, 
    output_dim=DENSEVEC_DIM, 
    input_length=1, 
    embeddings_constraint = unit_norm(),
    name='shared_embedding')
embedded_w1 = shared_embedding(word1)
embedded_w2 = shared_embedding(word2)
w1 = Flatten()(embedded_w1)
w2 = Flatten()(embedded_w2)
dotted = Dot(axes=1, name='dot_product')([w1, w2])
prediction = Dense(1, activation='sigmoid', name='output_layer')(dotted)
sg_model = Model(inputs=[word1, word2], outputs=prediction)
```
fastText
```python
ft_model = Sequential()
ft_model.add(Embedding(
    input_dim = MAX_FEATURES,
    output_dim = EMBEDDING_DIMS,
    input_length= MAXLEN))
ft_model.add(GlobalAveragePooling1D())
ft_model.add(Dense(1, activation='sigmoid'))
```
Models and Training data construction
The first step for CBOW and skip-gram
Our training corpus is a collection of sentences, Tweets, emails, comments, or even longer documents.  It is something composed of words.  Each word takes is turn being the "target" word, and we collect the n words behind it and n words which follow it.  This n is referred to as window size.  If our example document is the sentence "I love deep learning" and the window size is 1, we'd get:
  * I, love
  * I, love, deep
  * love, deep, learning
  * deep, learning
The target word is bold.
Skip-gram model training data
Skip-gram means form word pairs with a target word and all words in the window.  These become the "positive" (1) samples for the skip-gram algorithm.  In our "I love deep learning" example we'd get (eliminating repeated pairs):
(I, love) = 1
(love, deep) = 1
(deep, learning) = 1
To create negative samples (0), we pair random vocabulary words with the target word.  Yes, it's possible to unluckily pick a negative sample that usually appears around the target word.
For our prediction task, we'll take the dot product of the words in each pair (a small step away from the cosine similarity).  The training will keep tweaking the word vectors to make this product as close to unity as possible for our positive samples, and zero for our negative samples.
Happily, Keras include a function for creating skipgrams from text.  It even does the negative sampling for us.
End of explanation
id2word = { wordid: word for word, wordid in word2id.items()}
id2word
encoded_text = [word2id[word] for word in text_to_word_sequence(text1)]
encoded_text
[word2id[word] for word in text_to_word_sequence(text2)]
sg = skipgrams(encoded_text, vocabulary_size=len(word2id.keys()), window_size=1)
sg
for i in range(len(sg[0])):
    print "({0},{1})={2}".format(id2word[sg[0][i][0]], id2word[sg[0][i][1]], sg[1][i])
Explanation: Note word id's are numbered from 1, not zero
End of explanation
VOCAB_SIZE = len(word2id.keys())
VOCAB_SIZE
DENSEVEC_DIM = 50
Explanation: Model parameters
End of explanation
import keras
from keras.layers.embeddings import Embedding
from keras.constraints import unit_norm
from keras.layers.merge import Dot
from keras.layers.core import Activation
from keras.layers.core import Flatten
from keras.layers import Input, Dense
from keras.models import Model
Explanation: Model build
End of explanation
word1 = Input(shape=(1,), dtype='int64', name='word1')
word2 = Input(shape=(1,), dtype='int64', name='word2')
shared_embedding = Embedding(
    input_dim=VOCAB_SIZE+1, 
    output_dim=DENSEVEC_DIM, 
    input_length=1, 
    embeddings_constraint = unit_norm(),
    name='shared_embedding')
embedded_w1 = shared_embedding(word1)
embedded_w2 = shared_embedding(word2)
w1 = Flatten()(embedded_w1)
w2 = Flatten()(embedded_w2)
dotted = Dot(axes=1, name='dot_product')([w1, w2])
prediction = Dense(1, activation='sigmoid', name='output_layer')(dotted)
sg_model = Model(inputs=[word1, word2], outputs=prediction)
sg_model.compile(optimizer='adam', loss='mean_squared_error')
Explanation: Create a dense vector for each word in the pair.  The output of Embedding has shape (batch_size, sequence_length, output_dim) which in our case is (batch_size, 1, DENSEVEC_DIM).  We'll use Flatten to get rid of that pesky middle dimension (1), so going into the dot product we'll have shape (batch_size, DENSEVEC_DIM).
End of explanation
sg_model.layers
def print_layer(model, num):
    print model.layers[num]
    print model.layers[num].input_shape
    print model.layers[num].output_shape
print_layer(sg_model,3)
Explanation: At this point you can check out how the data flows through your compiled model.
End of explanation
import numpy as np
pairs = np.array(sg[0])
targets = np.array(sg[1])
targets
pairs
w1_list = np.reshape(pairs[:, 0], (len(pairs), 1))
w1_list
w2_list = np.reshape(pairs[:, 1], (len(pairs), 1))
w2_list
w2_list.shape
w2_list.dtype
sg_model.fit(x=[w1_list, w2_list], y=targets,  epochs=10)
sg_model.layers[2].weights
Explanation: Let's try training it with our toy data set!
End of explanation
MAX_FEATURES = 20000  # number of unique words in the dataset
MAXLEN = 400  # max word (feature) length of a review  
EMBEDDING_DIMS = 50
NGRAM_RANGE = 2
Explanation: Continuous Bag of Words (CBOW) model
CBOW means we take all the words in the window and use them to predict the target word.  Note we are trying to predict an actual word (or a probability distribution over words) with CBOW, whereas in skip-gram we are trying to predict a similarity score. 
FastText Model
FastText is creating dense document vectors using the words in the document enhanced with n-grams.  These are embedded, averaged, and fed through a hidden dense layer, with a sigmoid activation.  The prediction task is some binary classification of the documents.  As usual, after training we can extract the dense vectors from the model.
FastText Model Data Prep
End of explanation
def create_ngram_set(input_list, ngram_value=2):
    
    Extract a set of n-grams from a list of integers.
    
    return set(zip(*[input_list[i:] for i in range(ngram_value)]))
create_ngram_set([1, 2, 3, 4, 5], ngram_value=2)
create_ngram_set([1, 2, 3, 4, 5], ngram_value=3)
def add_ngram(sequences, token_indice, ngram_range=2):
    
    Augment the input list of list (sequences) by appending n-grams values.
    
    new_sequences = []
    for input_list in sequences:
        new_list = input_list[:]
        for i in range(len(new_list) - ngram_range + 1):
            for ngram_value in range(2, ngram_range + 1):
                ngram = tuple(new_list[i:i + ngram_value])
                if ngram in token_indice:
                    new_list.append(token_indice[ngram])
        new_sequences.append(new_list)
    return new_sequences
sequences = [[1,2,3,4,5, 6], [6,7,8]]
token_indice = {(1,2): 20000, (4,5): 20001, (6,7,8): 20002}
add_ngram(sequences, token_indice, ngram_range=2)
add_ngram(sequences, token_indice, ngram_range=3)
Explanation: Some data prep functions lifted from the example
End of explanation
from keras.datasets import imdb
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=MAX_FEATURES)
x_train[0:2]
y_train[0:2]
Explanation: load canned training data
End of explanation
ngram_set = set()
for input_list in x_train:
    for i in range(2, NGRAM_RANGE + 1):
        set_of_ngram = create_ngram_set(input_list, ngram_value=i)
        ngram_set.update(set_of_ngram)
len(ngram_set)
Explanation: Add n-gram features
End of explanation
ngram_set.pop()
start_index = MAX_FEATURES + 1
token_indice = {v: k + start_index for k, v in enumerate(ngram_set)}
indice_token = {token_indice[k]: k for k in token_indice}
Explanation: Assign id's to the new features
End of explanation
import numpy as np
MAX_FEATURES = np.max(list(indice_token.keys())) + 1
MAX_FEATURES
Explanation: Update MAX_FEATURES
End of explanation
x_train = add_ngram(x_train, token_indice, NGRAM_RANGE)
x_test = add_ngram(x_test, token_indice, NGRAM_RANGE)
Explanation: Add n-grams to the input data
End of explanation
from keras.preprocessing import sequence
sequence.pad_sequences([[1,2,3,4,5], [6,7,8]], maxlen=10)
x_train = sequence.pad_sequences(x_train, maxlen=MAXLEN)
x_test = sequence.pad_sequences(x_test, maxlen=MAXLEN)
x_train.shape
x_test.shape
Explanation: Make all input sequences the same length by padding with zeros
End of explanation
  Image('diagrams/fasttext.png')
from keras.models import Sequential
from keras.layers.embeddings import Embedding
from keras.layers.pooling import GlobalAveragePooling1D
from keras.layers import Dense
ft_model = Sequential()
ft_model.add(Embedding(
    input_dim = MAX_FEATURES,
    output_dim = EMBEDDING_DIMS,
    input_length= MAXLEN))
ft_model.add(GlobalAveragePooling1D())
ft_model.add(Dense(1, activation='sigmoid'))
ft_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
ft_model.layers
print_layer(ft_model, 0)
print_layer(ft_model, 1)
print_layer(ft_model, 2)
ft_model.fit(x_train, y_train, batch_size=100, epochs=3, validation_data=(x_test, y_test))
Explanation: FastText Model
End of explanation
Image('diagrams/text-cnn-classifier.png')
Explanation: fastText classifier vs. convolutional neural network (CNN) vs. long short-term memory (LSTM) classifier: Fight!
A CNN takes the dot product of various "filters" (some new vector) with each word window down the sentence.  For each convolutional layer in your model, you can choose the size of the filter (for example, 3 word vectors long) and the number of filters in the layer (for example, ten 3-word filters, or five 3-word filters).  
Add a bias to each dot product of the filter and word window, and run it through an activation function.  This produces a number.
Running a single filter down a sentence produces a series of numbers.  Generally the maximum value is taken to represent the alignment of the sentence with that particular filter.  All of this is just another way of extracting features from a sentence.  In fastText, we extracted features in a human-readable way (n-grams) and tacked them onto the input data.  With a CNN we take a different approach, letting the algorithm figure out what makes good features for the dataset.
insert filter operating on sentence image here
End of explanation
embedding_dim = 50  # we'll get a vector representation of words as a by-product
filter_sizes = (2, 3, 4)  # we'll make one convolutional layer for each filter we specify here
num_filters = 10  # each layer will contain this many filters
dropout_prob = (0.2, 0.2)
hidden_dims = 50
# Prepossessing parameters
sequence_length = 400
max_words = 5000
Explanation: Diagram from Convolutional Neural Networks for Sentence Classification, Kim Yoon (2014)
A CNN sentence classifier
End of explanation
from keras.datasets import imdb
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_words)  # limits vocab to num_words
?imdb.load_data
from keras.preprocessing import sequence
x_train = sequence.pad_sequences(x_train, maxlen=sequence_length, padding="post", truncating="post")
x_test = sequence.pad_sequences(x_test, maxlen=sequence_length, padding="post", truncating="post")
x_train[0]
vocabulary = imdb.get_word_index()  # word to integer map
vocabulary['good']
len(vocabulary)
Explanation: Canned input data
End of explanation
from keras.models import Model
from keras.layers import Input
from keras.layers import Embedding
from keras.layers import Dropout
from keras.layers import Conv1D
from keras.layers import MaxPooling1D
from keras.layers import Flatten
from keras.layers import Dense
from keras.layers.merge import Concatenate
# Input, embedding, and dropout layers
input_shape = (sequence_length,)
model_input = Input(shape=input_shape)
z = Embedding(
        input_dim=len(vocabulary) + 1, 
        output_dim=embedding_dim, 
        input_length=sequence_length, 
        name="embedding")(model_input)
z = Dropout(dropout_prob[0])(z)
# Convolutional block
# parallel set of n convolutions; output of all n are
# concatenated into one vector
conv_blocks = []
for sz in filter_sizes:
    conv = Conv1D(filters=num_filters, kernel_size=sz, activation="relu" )(z)
    conv = MaxPooling1D(pool_size=2)(conv)
    conv = Flatten()(conv)
    conv_blocks.append(conv)
    
z = Concatenate()(conv_blocks) if len(conv_blocks) > 1 else conv_blocks[0]
z = Dropout(dropout_prob[1])(z)
# Hidden dense layer and output layer
z = Dense(hidden_dims, activation="relu")(z)
model_output = Dense(1, activation="sigmoid")(z)
cnn_model = Model(model_input, model_output)
cnn_model.compile(loss="binary_crossentropy", optimizer="adam", metrics=["accuracy"])
cnn_model.layers
print_layer(cnn_model, 12)
print_layer(cnn_model, 12)
cnn_model.fit(x_train, y_train, batch_size=64, epochs=3, validation_data=(x_test, y_test))
cnn_model.layers[1].weights
cnn_model.layers[1].get_weights()
cnn_model.layers[3].weights
Explanation: Model build
End of explanation
Image('diagrams/LSTM.png')
from keras.models import Sequential
from keras.layers import Embedding
from keras.layers.core import SpatialDropout1D
from keras.layers.core import Dropout
from keras.layers.recurrent import LSTM
from keras.layers.core import Dense
hidden_dims = 50
embedding_dim = 50
lstm_model = Sequential()
lstm_model.add(Embedding(len(vocabulary) + 1, embedding_dim, input_length=sequence_length, name="embedding"))
lstm_model.add(SpatialDropout1D(Dropout(0.2)))
lstm_model.add(LSTM(hidden_dims, dropout=0.2, recurrent_dropout=0.2))  # first arg, like Dense, is dim of output
lstm_model.add(Dense(1, activation='sigmoid'))
lstm_model.compile(loss="binary_crossentropy", optimizer="adam", metrics=["accuracy"])
lstm_model.fit(x_train, y_train, batch_size=64, epochs=3, validation_data=(x_test, y_test))
lstm_model.layers
lstm_model.layers[2].input_shape
lstm_model.layers[2].output_shape
Explanation: An LSTM sentence classifier
End of explanation
%matplotlib inline
import pandas as pd
import glob
datapath = "/Users/pfigliozzi/aclImdb/train/unsup"
files = glob.glob(datapath+"/*.txt")[:1000] #first 1000 (there are 50k)
df = pd.concat([pd.read_table(filename, header=None, names=['raw']) for filename in files], ignore_index=True)  
df.raw.map(lambda x: len(x)).plot.hist()
50000. * 2000. / 10**6
Explanation: Appendix: Our own data download and preparation
We'll use the Large Movie Review Dataset v1.0 for our corpus.  While Keras has its own data samples you can import for modeling (including this one), I think it's very important to get and process your own data.  Otherwise, the results appear to materialize out of thin air and it's more difficult to get on with your own research.
End of explanation
<END_TASK>