<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Feature
Step1:  Config
Automatically discover the paths to various data folders and compose the project structure.
Step2:  Identifier for storing these features on disk and referring to them later.
Step3:  Make subsequent NN runs reproducible.
Step4:  Read data
Word embedding lookup matrix.
Step5:  Padded sequences of word indices for every question.
Step6:  Magic features.
Step7:  Word embedding properties.
Step9:  Define models
Step10:  Partition the data
Step11:  Create placeholders for out-of-fold predictions.
Step12:  Define hyperparameters
Step13:  Best values picked by Bayesian optimization.
Step14:  The path where the best weights of the current model will be saved.
Step15:  Fit the folds and compute out-of-fold predictions
Step16:  Save features
Step17:  Explore
<ASSISTANT_TASK:>
Python Code:
from pygoose import *
import gc
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import *
from keras import backend as K
from keras.models import Model, Sequential
from keras.layers import *
from keras.callbacks import EarlyStopping, ModelCheckpoint
Explanation: Feature: Out-Of-Fold Predictions from a Bidirectional LSTM (+Magic Inputs)
In addition to the bidirectional LSTM architecture, we'll append some of the leaky features to the intermediate feature layer.
<img src="assets/bi-lstm-with-magic.png" alt="Network Architecture" style="height: 700px;" />
Imports
This utility package imports numpy, pandas, matplotlib and a helper kg module into the root namespace.
End of explanation
project = kg.Project.discover()
Explanation: Config
Automatically discover the paths to various data folders and compose the project structure.
End of explanation
feature_list_id = 'oofp_nn_bi_lstm_with_magic'
Explanation: Identifier for storing these features on disk and referring to them later.
End of explanation
RANDOM_SEED = 42
np.random.seed(RANDOM_SEED)
Explanation: Make subsequent NN runs reproducible.
End of explanation
embedding_matrix = kg.io.load(project.aux_dir + 'fasttext_vocab_embedding_matrix.pickle')
Explanation: Read data
Word embedding lookup matrix.
End of explanation
X_train_q1 = kg.io.load(project.preprocessed_data_dir + 'sequences_q1_fasttext_train.pickle')
X_train_q2 = kg.io.load(project.preprocessed_data_dir + 'sequences_q2_fasttext_train.pickle')
X_test_q1 = kg.io.load(project.preprocessed_data_dir + 'sequences_q1_fasttext_test.pickle')
X_test_q2 = kg.io.load(project.preprocessed_data_dir + 'sequences_q2_fasttext_test.pickle')
y_train = kg.io.load(project.features_dir + 'y_train.pickle')
Explanation: Padded sequences of word indices for every question.
End of explanation
magic_feature_lists = [
    'magic_frequencies',
    'magic_cooccurrence_matrix',
]
X_train_magic, X_test_magic, _ = project.load_feature_lists(magic_feature_lists)
X_train_magic = X_train_magic.values
X_test_magic = X_test_magic.values
scaler = StandardScaler()
scaler.fit(np.vstack([X_train_magic, X_test_magic]))
X_train_magic = scaler.transform(X_train_magic)
X_test_magic = scaler.transform(X_test_magic)
Explanation: Magic features.
End of explanation
EMBEDDING_DIM = embedding_matrix.shape[-1]
VOCAB_LENGTH = embedding_matrix.shape[0]
MAX_SEQUENCE_LENGTH = X_train_q1.shape[-1]
print(EMBEDDING_DIM, VOCAB_LENGTH, MAX_SEQUENCE_LENGTH)
Explanation: Word embedding properties.
End of explanation
def create_model_question_branch():
    input_q = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
    
    embedding_q = Embedding(
        VOCAB_LENGTH,
        EMBEDDING_DIM,
        weights=[embedding_matrix],
        input_length=MAX_SEQUENCE_LENGTH,
        trainable=False,
    )(input_q)
    timedist_q = TimeDistributed(Dense(
        EMBEDDING_DIM,
        activation='relu',
    ))(embedding_q)
    lambda_q = Lambda(
        lambda x: K.max(x, axis=1),
        output_shape=(EMBEDDING_DIM, )
    )(timedist_q)
    
    output_q = lambda_q
    return input_q, output_q
def create_model(params):    
    embedding_layer = Embedding(
        VOCAB_LENGTH,
        EMBEDDING_DIM,
        weights=[embedding_matrix],
        input_length=MAX_SEQUENCE_LENGTH,
        trainable=False,
    )
    lstm_layer = Bidirectional(LSTM(
        params['num_lstm'],
        dropout=params['lstm_dropout_rate'],
        recurrent_dropout=params['lstm_dropout_rate'],
    ))
    sequence_1_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
    embedded_sequences_1 = embedding_layer(sequence_1_input)
    x1 = lstm_layer(embedded_sequences_1)
    sequence_2_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')
    embedded_sequences_2 = embedding_layer(sequence_2_input)
    y1 = lstm_layer(embedded_sequences_2)
    magic_input = Input(shape=(X_train_magic.shape[-1], ))
    
    merged = concatenate([x1, y1, magic_input])
    merged = Dropout(params['dense_dropout_rate'])(merged)
    merged = BatchNormalization()(merged)
    merged = Dense(params['num_dense'], activation='relu')(merged)
    merged = Dropout(params['dense_dropout_rate'])(merged)
    merged = BatchNormalization()(merged)
    output = Dense(1, activation='sigmoid')(merged)
    model = Model(
        inputs=[sequence_1_input, sequence_2_input, magic_input],
        outputs=output
    )
    model.compile(
        loss='binary_crossentropy',
        optimizer='nadam',
        metrics=['accuracy']
    )
    return model
def predict(model, X_q1, X_q2, X_magic):
    
    Mirror the pairs, compute two separate predictions, and average them.
    
    
    y1 = model.predict([X_q1, X_q2, X_magic], batch_size=1024, verbose=1).reshape(-1)   
    y2 = model.predict([X_q2, X_q1, X_magic], batch_size=1024, verbose=1).reshape(-1)    
    return (y1 + y2) / 2
Explanation: Define models
End of explanation
NUM_FOLDS = 5
kfold = StratifiedKFold(
    n_splits=NUM_FOLDS,
    shuffle=True,
    random_state=RANDOM_SEED
)
Explanation: Partition the data
End of explanation
y_train_oofp = np.zeros_like(y_train, dtype='float64')
y_test_oofp = np.zeros((len(X_test_q1), NUM_FOLDS))
Explanation: Create placeholders for out-of-fold predictions.
End of explanation
BATCH_SIZE = 2048
MAX_EPOCHS = 200
Explanation: Define hyperparameters
End of explanation
model_params = {
    'dense_dropout_rate': 0.075,
    'lstm_dropout_rate': 0.332,
    'num_dense': 130,
    'num_lstm': 300,
}
Explanation: Best values picked by Bayesian optimization.
End of explanation
model_checkpoint_path = project.temp_dir + 'fold-checkpoint-' + feature_list_id + '.h5'
Explanation: The path where the best weights of the current model will be saved.
End of explanation
%%time
# Iterate through folds.
for fold_num, (ix_train, ix_val) in enumerate(kfold.split(X_train_q1, y_train)):
    
    # Augment the training set by mirroring the pairs.
    X_fold_train_q1 = np.vstack([X_train_q1[ix_train], X_train_q2[ix_train]])
    X_fold_train_q2 = np.vstack([X_train_q2[ix_train], X_train_q1[ix_train]])
    X_fold_train_magic = np.vstack([X_train_magic[ix_train], X_train_magic[ix_train]])
    X_fold_val_q1 = np.vstack([X_train_q1[ix_val], X_train_q2[ix_val]])
    X_fold_val_q2 = np.vstack([X_train_q2[ix_val], X_train_q1[ix_val]])
    X_fold_val_magic = np.vstack([X_train_magic[ix_val], X_train_magic[ix_val]])
    # Ground truth should also be "mirrored".
    y_fold_train = np.concatenate([y_train[ix_train], y_train[ix_train]])
    y_fold_val = np.concatenate([y_train[ix_val], y_train[ix_val]])
    
    print()
    print(f'Fitting fold {fold_num + 1} of {kfold.n_splits}')
    print()
    
    # Compile a new model.
    model = create_model(model_params)
    # Train.
    model.fit(
        [X_fold_train_q1, X_fold_train_q2, X_fold_train_magic], y_fold_train,
        validation_data=([X_fold_val_q1, X_fold_val_q2, X_fold_val_magic], y_fold_val),
        batch_size=BATCH_SIZE,
        epochs=MAX_EPOCHS,
        verbose=1,
        
        callbacks=[
            # Stop training when the validation loss stops improving.
            EarlyStopping(
                monitor='val_loss',
                min_delta=0.001,
                patience=3,
                verbose=1,
                mode='auto',
            ),
            # Save the weights of the best epoch.
            ModelCheckpoint(
                model_checkpoint_path,
                monitor='val_loss',
                save_best_only=True,
                verbose=2,
            ),
        ],
    )
        
    # Restore the best epoch.
    model.load_weights(model_checkpoint_path)
    
    # Compute out-of-fold predictions.
    y_train_oofp[ix_val] = predict(model, X_train_q1[ix_val], X_train_q2[ix_val], X_train_magic[ix_val])
    y_test_oofp[:, fold_num] = predict(model, X_test_q1, X_test_q2, X_test_magic)
    
    # Clear GPU memory.
    K.clear_session()
    del X_fold_train_q1, X_fold_train_q2, X_fold_train_magic
    del X_fold_val_q1, X_fold_val_q2, X_fold_val_magic
    del model
    gc.collect()
cv_score = log_loss(y_train, y_train_oofp)
print('CV score:', cv_score)
Explanation: Fit the folds and compute out-of-fold predictions
End of explanation
feature_names = [feature_list_id]
features_train = y_train_oofp.reshape((-1, 1))
features_test = np.mean(y_test_oofp, axis=1).reshape((-1, 1))
project.save_features(features_train, features_test, feature_names, feature_list_id)
Explanation: Save features
End of explanation
pd.DataFrame(features_test).plot.hist()
Explanation: Explore
End of explanation
<END_TASK>