<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Preprocessing
Step1:  Hide all GPUs from TensorFlow to not automatically occupy any GPU RAM.
Step2:  Config
Automatically discover the paths to various data folders and compose the project structure.
Step3:  The maximum allowed size of the embedding matrix and the maximum length our sequences will be padded/trimmed to.
Step4:  Load data
Preprocessed and tokenized questions. Stopwords should be kept for neural models.
Step5:  Word embedding database queried from the trained FastText model.
Step6:  Build features
Collect all texts
Step7:  Create question sequences
Step8:  Create embedding lookup matrix
Step9:  Allocate an embedding matrix. Include the NULL word.
Step10:  Fill the matrix using the vectors for individual words.
Step11:  Save features
Word embedding lookup matrix.
Step12:  Padded word index sequences.
<ASSISTANT_TASK:>
Python Code:
from pygoose import *
from gensim.models.wrappers.fasttext import FastText
Explanation: Preprocessing: FastText Sequences & Embeddings
Based on the tokenized questions and a pre-built word embedding database, build fixed-length (padded) sequences of word indices for each question, as well as a lookup matrix that maps word indices to word vectors.
Imports
This utility package imports numpy, pandas, matplotlib and a helper kg module into the root namespace.
End of explanation
kg.gpu.cuda_disable_gpus()
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
Explanation: Hide all GPUs from TensorFlow to not automatically occupy any GPU RAM.
End of explanation
project = kg.Project.discover()
Explanation: Config
Automatically discover the paths to various data folders and compose the project structure.
End of explanation
MAX_VOCAB_SIZE = 125000
MAX_SEQUENCE_LENGTH = 30
Explanation: The maximum allowed size of the embedding matrix and the maximum length our sequences will be padded/trimmed to.
End of explanation
tokens_train = kg.io.load(project.preprocessed_data_dir + 'tokens_lowercase_spellcheck_train.pickle')
tokens_test = kg.io.load(project.preprocessed_data_dir + 'tokens_lowercase_spellcheck_test.pickle')
Explanation: Load data
Preprocessed and tokenized questions. Stopwords should be kept for neural models.
End of explanation
embedding_model = FastText.load_word2vec_format(project.aux_dir + 'fasttext_vocab.vec')
EMBEDDING_DIM = len(embedding_model['apple'])
Explanation: Word embedding database queried from the trained FastText model.
End of explanation
texts_q1_train = [' '.join(pair[0]) for pair in tokens_train]
texts_q2_train = [' '.join(pair[1]) for pair in tokens_train]
texts_q1_test = [' '.join(pair[0]) for pair in tokens_test]
texts_q2_test = [' '.join(pair[1]) for pair in tokens_test]
unique_question_texts = list(set(texts_q1_train + texts_q2_train + texts_q1_test + texts_q2_test))
Explanation: Build features
Collect all texts
End of explanation
tokenizer = Tokenizer(
    num_words=MAX_VOCAB_SIZE,
    split=' ',
    lower=True,
    char_level=False,
)
tokenizer.fit_on_texts(unique_question_texts)
sequences_q1_train = tokenizer.texts_to_sequences(texts_q1_train)
sequences_q2_train = tokenizer.texts_to_sequences(texts_q2_train)
sequences_q1_test = tokenizer.texts_to_sequences(texts_q1_test)
sequences_q2_test = tokenizer.texts_to_sequences(texts_q2_test)
Explanation: Create question sequences
End of explanation
num_words = min(MAX_VOCAB_SIZE, len(tokenizer.word_index))
Explanation: Create embedding lookup matrix
End of explanation
embedding_matrix = np.zeros((num_words + 1, EMBEDDING_DIM))
Explanation: Allocate an embedding matrix. Include the NULL word.
End of explanation
for word, index in progressbar(tokenizer.word_index.items()):
    if word in embedding_model.vocab:
        embedding_matrix[index] = embedding_model[word]
Explanation: Fill the matrix using the vectors for individual words.
End of explanation
kg.io.save(embedding_matrix, project.aux_dir + 'fasttext_vocab_embedding_matrix.pickle')
Explanation: Save features
Word embedding lookup matrix.
End of explanation
sequences_q1_padded_train = pad_sequences(sequences_q1_train, maxlen=MAX_SEQUENCE_LENGTH)
sequences_q2_padded_train = pad_sequences(sequences_q2_train, maxlen=MAX_SEQUENCE_LENGTH)
sequences_q1_padded_test = pad_sequences(sequences_q1_test, maxlen=MAX_SEQUENCE_LENGTH)
sequences_q2_padded_test = pad_sequences(sequences_q2_test, maxlen=MAX_SEQUENCE_LENGTH)
kg.io.save(sequences_q1_padded_train, project.preprocessed_data_dir + 'sequences_q1_fasttext_train.pickle')
kg.io.save(sequences_q2_padded_train, project.preprocessed_data_dir + 'sequences_q2_fasttext_train.pickle')
kg.io.save(sequences_q1_padded_test, project.preprocessed_data_dir + 'sequences_q1_fasttext_test.pickle')
kg.io.save(sequences_q2_padded_test, project.preprocessed_data_dir + 'sequences_q2_fasttext_test.pickle')
Explanation: Padded word index sequences.
End of explanation
<END_TASK>