<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Helpers
Step1:  Load Annotated Data
Step2:  Params
Step3:  Prep Data
Step4:  Sklearn Experiments
Lets run some quick experiments in sklearn, so that we have baselines for the following models built in keras. We will only be building logistic regressions with one-hot labels. This will also help us see if we should use tfidf weighting and normalization.
Step5:  No tfidf
Step6:  With tfidf
Step7:  TFIDF improves the ROC score for both types of ngram models although it gives a bigger boost for the char-ngram models.
Tensorflow/Keras
Now we will cross-validate over model architectures (linear, mlp, lstm), ngram type (word, char), and label type (one hot or empirical distribution)
Linear and MLP
The mlp model class actually includes linear models (just set hidden layers to be empty)
Step8:  LSTM
Step9:  Conv LSTM
<ASSISTANT_TASK:>
Python Code:
def get_best_estimator(cv):
    params = cv.best_params_
    model = cv.estimator
    model = model.set_params(**params)
    return model
    
def save_best_estimator(cv, directory, name):
    model = get_best_estimator(cv)
    save_pipeline(model, directory, name)
Explanation: Helpers
End of explanation
task = 'attack'
data = load_comments_and_labels(task)
Explanation: Load Annotated Data
End of explanation
path = '../../models/cv/'
n_max = 10000000
n_iter = 15
Explanation: Params
End of explanation
X_train, y_train_ohv = assemble_data(data, 'comments', 'plurality', splits = ['train'])
X_dev, y_dev_ohv = assemble_data(data,  'comments', 'plurality', splits = ['dev'])
_, y_train_ed = assemble_data(data, 'comments', 'empirical_dist', splits = ['train'])
_, y_dev_ed = assemble_data(data,  'comments', 'empirical_dist', splits = ['dev'])
y_train_ohm = one_hot(y_train_ed)
y_dev_ohm = one_hot(y_dev_ed)
X_train = X_train[:n_max]
X_dev = X_dev[:n_max]
y_train_ohv = y_train_ohv[:n_max]
y_dev_ohv = y_dev_ohv[:n_max]
y_train_ed = y_train_ed[:n_max]
y_dev_ed = y_dev_ed[:n_max]
y_train_ohm = y_train_ohm[:n_max]
y_dev_ohm = y_dev_ohm[:n_max]
results_list = []
Explanation: Prep Data
End of explanation
max_features = (5000, 10000, 50000, 100000)
C = (0.0001, 0.001, 0.01, 0.1, 1, 10)
Explanation: Sklearn Experiments
Lets run some quick experiments in sklearn, so that we have baselines for the following models built in keras. We will only be building logistic regressions with one-hot labels. This will also help us see if we should use tfidf weighting and normalization.
End of explanation
alg = Pipeline([
    ('vect', CountVectorizer()),
    ('clf', LogisticRegression()),
])
# linear char-gram, no tfidf
param_grid = {
    'vect__max_features': max_features, 
    'vect__ngram_range': ((1,5),),  
    'vect__analyzer' : ('char',),
    'clf__C' : C,
}
m = tune (X_train, y_train_ohv, X_dev, y_dev_ohv, alg, param_grid, n_iter, roc_scorer, n_jobs = 6, verbose = True)
# linear word-gram, no tfidf
param_grid = {
    'vect__max_features': max_features, 
    'vect__ngram_range': ((1,2),),  
    'vect__analyzer' : ('word',),
    'clf__C' : C,
}
m = tune (X_train, y_train_ohv, X_dev, y_dev_ohv, alg, param_grid, n_iter, roc_scorer, n_jobs = 6, verbose = True)
Explanation: No tfidf
End of explanation
alg = Pipeline([
    ('vect', CountVectorizer()),
    ('tfidf', TfidfTransformer()),
    ('clf', LogisticRegression()),
])
# linear char-gram, tfidf
param_grid = {
    'vect__max_features': max_features, 
    'vect__ngram_range': ((1,5),),  
    'vect__analyzer' : ('char',),
    'tfidf__sublinear_tf' : (True, False),
    'tfidf__norm' : (None, 'l2'),
    'clf__C' : C,
}
m = tune (X_train, y_train_ohv, X_dev, y_dev_ohv, alg, param_grid, n_iter, roc_scorer, n_jobs = 6, verbose = True)
# linear word-gram, tfidf
param_grid = {
    'vect__max_features': max_features, 
    'vect__ngram_range': ((1,2),),  
    'vect__analyzer' : ('word',),
    'tfidf__sublinear_tf' : (True, False),
    'tfidf__norm' : (None, 'l2'),
    'clf__C' : C,
}
m = tune (X_train, y_train_ohv, X_dev, y_dev_ohv, alg, param_grid, n_iter, roc_scorer, n_jobs = 6, verbose = True)
Explanation: With tfidf
End of explanation
alg = Pipeline([
    ('vect', CountVectorizer()),
    ('tfidf', TfidfTransformer()),
    ('to_dense', DenseTransformer()), 
    ('clf', KerasClassifier(build_fn=make_mlp, output_dim = 2, verbose=False)),
])
dependencies = [( 'vect__max_features', 'clf__input_dim')]
char_vec_params = {
    'vect__max_features': (5000, 10000, 30000), 
    'vect__ngram_range': ((1,5),),  
    'vect__analyzer' : ('char',)
    }
word_vect_params = {
    'vect__max_features': (5000, 10000, 30000), 
    'vect__ngram_range': ((1,2),),  
    'vect__analyzer' : ('word',)
    }
tfidf_params = {
    'tfidf__sublinear_tf' : (True, False),
    'tfidf__norm' : ('l2',),
    }
linear_clf_params = {
    'clf__alpha' : (0.000000001, 0.0000001, 0.00001, 0.001, 0.01),
    'clf__hidden_layer_sizes' : ((),),
    'clf__nb_epoch' : (2,4,8,16),
    'clf__batch_size': (200,)
    }
mlp_clf_params = {
    'clf__alpha' : (0.000000001, 0.0000001, 0.00001, 0.001, 0.01),
    'clf__hidden_layer_sizes' : ((50,), (50, 50), (50, 50, 50)),
    'clf__nb_epoch' : (2,4,8,16),
    'clf__batch_size': (200,)
    }
for model in ['linear', 'mlp']:
    for gram in ['word', 'char']:
        for label in ['oh', 'ed']:
            params = {}
            
            if model == 'linear':
                params.update(linear_clf_params)
            else:
                params.update(mlp_clf_params)
                
            params.update(tfidf_params)
                
            if gram == 'char':
                params.update(char_vec_params)
            else:
                params.update(word_vect_params)
                
            if label == 'oh':
                y_train = y_train_ohm
                y_dev = y_dev_ohm
            else:
                y_train = y_train_ed
                y_dev = y_dev_ed
            
            print('\n\n\n %s %s %s' % (model, gram, label))
            cv = tune (X_train, y_train, X_dev, y_dev,
                      alg, params,
                      n_iter,
                      roc_scorer,
                      n_jobs = 1,
                      verbose = True,
                      dependencies = dependencies)
            
            save_best_estimator(cv, path, '%s_%s_%s' % (model, gram, label))
            est = get_best_estimator(cv)
            est.fit(X_train, y_train)
            
            best_spearman = spearman_scorer(est, X_dev, y_dev_ed) * 100
            print ("\n best spearman: ", best_spearman)
            best_roc = max(cv.grid_scores_, key=lambda x: x[1])[1] * 100
            print ("\n best roc: ", best_roc)
            
            results_list.append({'model_type': model,
                                 'ngram_type': gram,
                                 'label_type' : label,
                                 'cv': cv.grid_scores_,
                                 'best_roc': round(best_roc, 3),
                                 'best_spearman': round(best_spearman, 3)
                                })
results_df = pd.DataFrame(results_list)
results_df
grid_scores[0].mean_validation_score
grid_scores = results_df['cv'][0]
max(grid_scores, key = lambda x: x.mean_validation_score).parameters
import json
def get_best_params(grid_scores):
    return json.dumps(max(grid_scores, key = lambda x: x.mean_validation_score).parameters)
results_df['best_params'] = results_df['cv'].apply(get_best_params)
results_df.to_csv('cv_results.csv')
Explanation: TFIDF improves the ROC score for both types of ngram models although it gives a bigger boost for the char-ngram models.
Tensorflow/Keras
Now we will cross-validate over model architectures (linear, mlp, lstm), ngram type (word, char), and label type (one hot or empirical distribution)
Linear and MLP
The mlp model class actually includes linear models (just set hidden layers to be empty)
End of explanation
alg = Pipeline([
    ('seq', SequenceTransformer()),
    ('clf', KerasClassifier(build_fn=make_lstm, output_dim = 2, verbose=True)),
])
dependencies = [( 'seq__max_features', 'clf__max_features'),
                ( 'seq__max_len', 'clf__max_len')]
word_seq_params = {
    'seq__max_features' : (5000, 10000, 30000),
    'seq__max_len' : (100, 200, 500),
    'seq__analyzer' : ('word',)
}
char_seq_params = {
    'seq__max_features' : (100,),
    'seq__max_len' : (200, 500, 1000),
    'seq__analyzer' : ('char',)
}
clf_params = {
    'clf__dropout' : (0.1, 0.2, 0.4),
    'clf__embedding_size' : (64, 128),
    'clf__lstm_output_size': (64, 128),
    'clf__nb_epoch' : (2,3,4),
    'clf__batch_size': (200,)
}
from pprint import pprint
model = 'lstm'
for gram in ['word', 'char']:
    for label in ['oh', 'ed']:
        params = {}
        params.update(clf_params)
        if gram == 'char':
            params.update(char_seq_params)
        else:
            params.update(word_seq_params)
        if label == 'oh':
            y_train = y_train_ohm
            y_dev = y_dev_ohm
        else:
            y_train = y_train_ed
            y_dev = y_dev_ed
            
        pprint(params)
        print('\n\n\n %s %s %s' % (model, gram, label))
        cv = tune (X_train, y_train, X_dev, y_dev,
                  alg, params,
                  n_iter,
                  roc_scorer,
                  n_jobs = 1,
                  verbose = True,
                  dependencies = dependencies)
        save_best_estimator(cv, path, '%s_%s_%s' % (model, gram, label))
        est = get_best_estimator(cv)
        est.fit(X_train, y_train)
        
        best_spearman = spearman_scorer(est, X_dev, y_dev_ed) * 100
        print ("\n best spearman: ", best_spearman)
        best_roc = max(cv.grid_scores_, key=lambda x: x[1])[1] * 100
        print ("\n best roc: ", best_roc)
        results_list.append({'model_type': model,
                             'ngram_type': gram,
                             'label_type' : label,
                             'cv': cv.grid_scores_,
                             'best_roc': round(best_roc, 3),
                             'best_spearman': round(best_spearman, 3)
                            })
Explanation: LSTM
End of explanation
alg = Pipeline([
    ('seq', SequenceTransformer()),
    ('clf', KerasClassifier(build_fn=make_conv_lstm, output_dim = 2, verbose=True)),
])
dependencies = [( 'seq__max_features', 'clf__max_features'),
                ( 'seq__max_len', 'clf__max_len')]
word_seq_params = {
    'seq__max_features' : (5000, 10000, 30000),
    'seq__max_len' : (100, 200, 500),
    'seq__analyzer' : ('word',),
    'clf__filter_length': (2, 4, 6),
    'clf__pool_length' : (2, 4, 6)
}
char_seq_params = {
    'seq__max_features' : (100,),
    'seq__max_len' : (200, 500, 1000),
    'seq__analyzer' : ('char',),
    'clf__filter_length': (5, 10, 15),
    'clf__pool_length' : (5, 10, 15)
}
clf_params = {
    'clf__dropout' : (0.1, 0.2, 0.4),
    'clf__embedding_size' : (64, 128),
    'clf__lstm_output_size': (64, 128),
    'clf__nb_epoch' : (2,3,4),
    'clf__batch_size': (200,),
    'clf__nb_filter' : (64, 128),
    
}
model = 'conv_lstm'
for gram in ['word', 'char']:
    for label in ['oh', 'ed']:
        params = {}
        params.update(clf_params)
        if gram == 'char':
            params.update(char_seq_params)
        else:
            params.update(word_seq_params)
        if label == 'oh':
            y_train = y_train_ohm
            y_dev = y_dev_ohm
        else:
            y_train = y_train_ed
            y_dev = y_dev_ed
            
        pprint(params)
        print('\n\n\n %s %s %s' % (model, gram, label))
        cv = tune (X_train, y_train, X_dev, y_dev,
                  alg, params,
                  n_iter,
                  roc_scorer,
                  n_jobs = 1,
                  verbose = True,
                  dependencies = dependencies)
        save_best_estimator(cv, path, '%s_%s_%s' % (model, gram, label))
        est = get_best_estimator(cv)
        est.fit(X_train, y_train)
        
        best_spearman = spearman_scorer(est, X_dev, y_dev_ed) * 100
        print ("\n best spearman: ", best_spearman)
        best_roc = max(cv.grid_scores_, key=lambda x: x[1])[1] * 100
        print ("\n best roc: ", best_roc)
        results_list.append({'model_type': model,
                             'ngram_type': gram,
                             'label_type' : label,
                             'cv': cv.grid_scores_,
                             'best_roc': round(best_roc, 3),
                             'best_spearman': round(best_spearman, 3)
                            })
Explanation: Conv LSTM
End of explanation
<END_TASK>