<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Parsing the imdb database
Step1:  Download data
Step2:  Load Database
Step3:  Transform each review into exactly 500 words having a vocabulary of 10000 words
Step4:  Split data into training and test sets and make sure to make this balanced
Step5:  Use Stratify to make it balanced
<ASSISTANT_TASK:>
Python Code:
# Based on
# https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/6.1-using-word-embeddings.ipynb
# https://machinelearningmastery.com/develop-word-embeddings-python-gensim/
import warnings
warnings.filterwarnings('ignore')
%matplotlib inline
%pylab inline
import tensorflow as tf
tf.logging.set_verbosity(tf.logging.ERROR)
print(tf.__version__)
Explanation: Parsing the imdb database
End of explanation
!ls -l C:/Users/olive/Development/data/aclImdb
!cat C:/Users/olive/Development/data/aclImdb/README
# !ls -l C:/Users/olive/Development/data/aclImdb/train
Explanation: Download data:
original imdb database from http://ai.stanford.edu/~amaas/data/sentiment/
data from https://www.imdb.com/
End of explanation
import os
imdb_dir = 'C:/Users/olive/Development/data/aclImdb'
train_dir = os.path.join(imdb_dir, 'train')
labels = []
texts = []
for label_type in ['neg', 'pos']:
    dir_name = os.path.join(train_dir, label_type)
    for fname in os.listdir(dir_name):
        if fname[-4:] == '.txt':
            f = open(os.path.join(dir_name, fname), encoding='UTF-8')
            texts.append(f.read())
            f.close()
            if label_type == 'neg':
                labels.append(0)
            else:
                labels.append(1)
len(texts)
texts[0]
# 0: neg, 1: pos
labels[0]
texts[15000]
labels[15000]
Explanation: Load Database
End of explanation
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
import numpy as np
maxlen = 500  # We will cut reviews after 500 words
max_words = 10000  # We will only consider the top 10,000 words in the dataset
tokenizer = Tokenizer(num_words=max_words)
tokenizer.fit_on_texts(texts)
# Tokenizer?
tokenizer.word_index
# tokenizer.texts_to_matrix?
binary_martix = tokenizer.texts_to_matrix(texts, mode='binary')
binary_martix
len(binary_martix)
len(binary_martix[0])
count_matrix = tokenizer.texts_to_matrix(texts, mode='count')
count_matrix
len(count_matrix)
len(binary_martix[0])
tfidf_matrix = tokenizer.texts_to_matrix(texts, mode='tfidf')
tfidf_matrix
# tokenizer.texts_to_sequences?
sequences = tokenizer.texts_to_sequences(texts)
len(sequences)
len(sequences[0])
sequences[0]
data = pad_sequences(sequences, maxlen=maxlen)
labels = np.asarray(labels)
print('Shape of data tensor:', data.shape)
print('Shape of label tensor:', labels.shape)
len(data[0])
# by convention zero is the empty word
data[0]
Explanation: Transform each review into exactly 500 words having a vocabulary of 10000 words
End of explanation
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(data, labels, test_size=0.2, random_state=42)
x_train.shape
y_train.shape
np.unique(y_train, return_counts=True)
np.unique(y_test, return_counts=True)
Explanation: Split data into training and test sets and make sure to make this balanced
End of explanation
x_train, x_test, y_train, y_test = train_test_split(data, labels, test_size=0.2, random_state=42, stratify=labels)
np.unique(y_train, return_counts=True)
np.unique(y_test, return_counts=True)
Explanation: Use Stratify to make it balanced
End of explanation
<END_TASK>