<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 # Table of Contents
<div class="toc" style="margin-top
Step1:  Feature importance from from linear model
Train the model and get reference score
Step2:  Get random permutation scores and compute feature importance
Step3:  With embeddings
Train the model
Step4:  Get feature importance
Step5:  Neural net with hidden layer and embeddings
This is our best configuration for one year of data.
Train the model
Step6:  Evaluate feature importance
Step7:  Compare feature importances
Step8:  Why is t2m_std not important?
Is there a good correlation between forecast error and raw ansemble spread?
Step9:  What about the correlation between the error and PP spread?
Step10:  Does the NN care at all about the fc spread?
<ASSISTANT_TASK:>
Python Code:
%load_ext autoreload
%autoreload 2
%matplotlib inline
from nn_src.imports import *
from nn_src.utils import get_datasets
#DATA_DIR = '/Users/stephanrasp/data/'
# DATA_DIR = '/scratch/srasp/ppnn_data/'
DATA_DIR = '/Volumes/SanDisk/data/ppnn_data/'
aux_train_set, aux_test_set = get_datasets(DATA_DIR, 'aux_15_16.pkl', ['2015-01-01', '2016-01-01'], aux=True)
n_features = aux_train_set.features.shape[1]; n_features
Explanation: # Table of Contents
<div class="toc" style="margin-top: 1em;"><ul class="toc-item" id="toc-level0"><li><span><a href="http://localhost:8888/notebooks/feature_importance.ipynb#Feature-importance-from-from-linear-model" data-toc-modified-id="Feature-importance-from-from-linear-model-1"><span class="toc-item-num">1&nbsp;&nbsp;</span>Feature importance from from linear model</a></span><ul class="toc-item"><li><span><a href="http://localhost:8888/notebooks/feature_importance.ipynb#Train-the-model-and-get-reference-score" data-toc-modified-id="Train-the-model-and-get-reference-score-1.1"><span class="toc-item-num">1.1&nbsp;&nbsp;</span>Train the model and get reference score</a></span></li><li><span><a href="http://localhost:8888/notebooks/feature_importance.ipynb#Get-random-permutation-scores-and-compute-feature-importance" data-toc-modified-id="Get-random-permutation-scores-and-compute-feature-importance-1.2"><span class="toc-item-num">1.2&nbsp;&nbsp;</span>Get random permutation scores and compute feature importance</a></span></li></ul></li><li><span><a href="http://localhost:8888/notebooks/feature_importance.ipynb#With-embeddings" data-toc-modified-id="With-embeddings-2"><span class="toc-item-num">2&nbsp;&nbsp;</span>With embeddings</a></span><ul class="toc-item"><li><span><a href="http://localhost:8888/notebooks/feature_importance.ipynb#Train-the-model" data-toc-modified-id="Train-the-model-2.1"><span class="toc-item-num">2.1&nbsp;&nbsp;</span>Train the model</a></span></li><li><span><a href="http://localhost:8888/notebooks/feature_importance.ipynb#Get-feature-importance" data-toc-modified-id="Get-feature-importance-2.2"><span class="toc-item-num">2.2&nbsp;&nbsp;</span>Get feature importance</a></span></li></ul></li><li><span><a href="http://localhost:8888/notebooks/feature_importance.ipynb#Neural-net-with-hidden-layer-and-embeddings" data-toc-modified-id="Neural-net-with-hidden-layer-and-embeddings-3"><span class="toc-item-num">3&nbsp;&nbsp;</span>Neural net with hidden layer and embeddings</a></span><ul class="toc-item"><li><span><a href="http://localhost:8888/notebooks/feature_importance.ipynb#Train-the-model" data-toc-modified-id="Train-the-model-3.1"><span class="toc-item-num">3.1&nbsp;&nbsp;</span>Train the model</a></span></li><li><span><a href="http://localhost:8888/notebooks/feature_importance.ipynb#Evaluate-feature-importance" data-toc-modified-id="Evaluate-feature-importance-3.2"><span class="toc-item-num">3.2&nbsp;&nbsp;</span>Evaluate feature importance</a></span></li></ul></li><li><span><a href="http://localhost:8888/notebooks/feature_importance.ipynb#Compare-feature-importances" data-toc-modified-id="Compare-feature-importances-4"><span class="toc-item-num">4&nbsp;&nbsp;</span>Compare feature importances</a></span></li><li><span><a href="http://localhost:8888/notebooks/feature_importance.ipynb#Why-is-t2m_std-not-important?" data-toc-modified-id="Why-is-t2m_std-not-important?-5"><span class="toc-item-num">5&nbsp;&nbsp;</span>Why is t2m_std not important?</a></span><ul class="toc-item"><li><span><a href="http://localhost:8888/notebooks/feature_importance.ipynb#Is-there-a-good-correlation-between-forecast-error-and-raw-ansemble-spread?" data-toc-modified-id="Is-there-a-good-correlation-between-forecast-error-and-raw-ansemble-spread?-5.1"><span class="toc-item-num">5.1&nbsp;&nbsp;</span>Is there a good correlation between forecast error and raw ansemble spread?</a></span></li><li><span><a href="http://localhost:8888/notebooks/feature_importance.ipynb#What-about-the-correlation-between-the-error-and-PP-spread?" data-toc-modified-id="What-about-the-correlation-between-the-error-and-PP-spread?-5.2"><span class="toc-item-num">5.2&nbsp;&nbsp;</span>What about the correlation between the error and PP spread?</a></span></li><li><span><a href="http://localhost:8888/notebooks/feature_importance.ipynb#Does-the-NN-care-at-all-about-the-fc-spread?" data-toc-modified-id="Does-the-NN-care-at-all-about-the-fc-spread?-5.3"><span class="toc-item-num">5.3&nbsp;&nbsp;</span>Does the NN care at all about the fc spread?</a></span></li></ul></li></ul></div>
Feature importance
The skill of our models certainly increases with auxiliary variables. Of course we would like to know which of the 38 variables we added actually helped the predictions.
Random permutation importance gives us some insight. For this we randomly permute one feature at a time from the test set and evaluate the decrease in skill.
End of explanation
fc_aux = build_fc_model(n_features, 2, compile=True, lr=0.02)
fc_aux.summary()
fc_aux.fit(aux_train_set.features, aux_train_set.targets, 1024, 30, verbose=0)
# Get the reference score from the last model we trained
ref_score = fc_aux.evaluate(aux_test_set.features, aux_test_set.targets, 4096, 0); ref_score
Explanation: Feature importance from from linear model
Train the model and get reference score
End of explanation
def eval_shuf(m, idx, emb=False):
    x_shuf = aux_test_set.features.copy()
    x_shuf[:, idx] = np.random.permutation(x_shuf[:, idx])
    x = [x_shuf, aux_test_set.cont_ids] if emb else x_shuf
    return m.evaluate(x, aux_test_set.targets, 4096, 0)
def perm_imp(m):
    scores = [eval_shuf(m, i) for i in range(len(aux_test_set.feature_names))]
    fimp = np.array(scores) - ref_score
    df = pd.DataFrame(columns=['Feature', 'Importance'])
    df['Feature'] = aux_test_set.feature_names; df['Importance'] = fimp
    return df
fimp_fc_aux = perm_imp(fc_aux)
fig, ax = plt.subplots(figsize=(12, 5))
sns.barplot(data=fimp_fc_aux, y='Importance', x='Feature', ax=ax)
plt.xticks(rotation=90);
Explanation: Get random permutation scores and compute feature importance
End of explanation
emb_size = 2
max_id = int(np.max([aux_train_set.cont_ids.max(), aux_test_set.cont_ids.max()]))
max_id
fc_aux_emb = build_emb_model(n_features, 2, [], emb_size, max_id, compile=True, lr=0.02)
fc_aux_emb.fit([aux_train_set.features, aux_train_set.cont_ids], aux_train_set.targets, 
               epochs=30, batch_size=1024, verbose=0);
ref_score = fc_aux_emb.evaluate([aux_test_set.features, aux_test_set.cont_ids], aux_test_set.targets, 4096, 0)
ref_score
Explanation: With embeddings
Train the model
End of explanation
def perm_imp_emb(m, ref):
    scores = [eval_shuf(m, i, True) for i in range(len(aux_test_set.feature_names))]
    ids_shuf = np.random.permutation(aux_test_set.cont_ids)
    scores += [m.evaluate([aux_test_set.features, ids_shuf], aux_test_set.targets, 4096, 0)]
    fimp = np.array(scores) - ref
    df = pd.DataFrame(columns=['Feature', 'Importance'])
    df['Feature'] = aux_test_set.feature_names + ['Embedding']; df['Importance'] = fimp
    return df
fimp_fc_aux_emb = perm_imp_emb(fc_aux_emb, ref_score)
fig, ax = plt.subplots(figsize=(12, 5))
sns.barplot(data=fimp_fc_aux_emb, y='Importance', x='Feature', ax=ax)
plt.xticks(rotation=90);
Explanation: Get feature importance
End of explanation
nn_aux_emb = build_emb_model(n_features, 2, [50], emb_size, max_id, compile=True, lr=0.01)
nn_aux_emb.fit([aux_train_set.features, aux_train_set.cont_ids], aux_train_set.targets, 
               epochs=30, batch_size=1024, verbose=0);
ref_score = nn_aux_emb.evaluate([aux_test_set.features, aux_test_set.cont_ids], aux_test_set.targets, 4096, 0)
ref_score
Explanation: Neural net with hidden layer and embeddings
This is our best configuration for one year of data.
Train the model
End of explanation
fimp_nn_aux_emb = perm_imp_emb(nn_aux_emb, ref_score)
fig, ax = plt.subplots(figsize=(12, 5))
sns.barplot(data=fimp_nn_aux_emb, y='Importance', x='Feature', ax=ax)
plt.xticks(rotation=90);
Explanation: Evaluate feature importance
End of explanation
fimp_fc_aux = fimp_fc_aux.append({'Feature': 'Embedding'}, ignore_index=True)
fimps = [fimp_fc_aux, fimp_fc_aux_emb, fimp_nn_aux_emb]
comb_df = pd.DataFrame(data=fimp_fc_aux['Feature']); comb_df.head()
comb_df['FCN-aux'] = fimp_fc_aux['Importance']
comb_df['FCN-aux-emb'] = fimp_fc_aux_emb['Importance']
comb_df['NN-aux-emb'] = fimp_nn_aux_emb['Importance']
comb_df['Mean importance'] = comb_df.iloc[:, 1:].mean(axis=1)
comb_df.sort_values('Mean importance', ascending=False, inplace=True)
comb_df.head()
len(fimp_fc_aux)
comb_df.sum()
comb_df.iloc[0, 1:] /= 10
comb_df['Feature'].iloc[0] = 't2m_mean / 10'
melt_df = comb_df.iloc[:15, :4].melt(id_vars='Feature', var_name='Experiment', value_name='Importance')
melt_df['Feature'] = [f.replace('_fc_', '_') for f in melt_df['Feature']]
sns.set_style('white')
sns.set_palette(['#1E90FF', '#B22222', '#228B22'])
fig, ax = plt.subplots(figsize=(8, 4))
sns.barplot(data = melt_df, x='Feature', y='Importance', hue='Experiment', ax=ax)
plt.xticks(rotation=90);
sns.despine()
plt.title('Feature importance');
fig.savefig('./feature-importance.pdf', bbox_inches='tight')
Explanation: Compare feature importances
End of explanation
fc_err = aux_test_set.features[:, 0] * aux_test_set.scale_factors[0] - aux_test_set.targets
ens_spread = aux_test_set.features[:, 1] * aux_test_set.scale_factors[1]
plt.scatter(np.abs(fc_err[::10]), ens_spread[::10], alpha=0.05, s=10);
np.corrcoef(np.abs(fc_err), ens_spread)[0, 1]
np.abs(fc_err).mean(), ens_spread.mean(), ens_spread.mean()/np.abs(fc_err).mean()
Explanation: Why is t2m_std not important?
Is there a good correlation between forecast error and raw ansemble spread?
End of explanation
preds = nn_aux_emb.predict([aux_test_set.features, aux_test_set.cont_ids], 4096, 0)
pp_spread = np.abs(preds[:, 1])
plt.scatter(np.abs(fc_err[::10]), pp_spread[::10], alpha=0.05, s=10);
np.corrcoef(np.abs(fc_err), pp_spread)[0, 1]
pp_spread.mean()/np.abs(fc_err).mean()
Explanation: What about the correlation between the error and PP spread?
End of explanation
plt.scatter(ens_spread[::10], pp_spread[::10], alpha=0.05, s=10);
np.corrcoef(ens_spread, pp_spread)[0, 1]
np.mean(pp_spread), np.mean(ens_spread)
Explanation: Does the NN care at all about the fc spread?
End of explanation
<END_TASK>