<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Examine sensitivity of current best model (Gear-Specific, Multi-Window 
Logistic Model with is-daylight) with respect to the various random
seeds that are used during training.
Random seed for model itself. Expect minimal effect
Random seed for splitting training / testing data. Perhaps larger effect
As it turns out (see below), neither makes much of a difference. Since this
test was originally convceived the MMSI selection became a lot less random
in order to better match the number of test and training MMSI. As a result,
this lack of sensitivity may not hold as we add new MMSI. That said, we can
expect the results to improve as we add more MMSI going forward.
Step1:  First investigate sensitivity of the LogisiticModels to different seeds
Step2:  Essentially no difference when setting the seed for different runs.
What about setting the seed when loading the data?
<ASSISTANT_TASK:>
Python Code:
%matplotlib inline
from vessel_scoring import data, utils
from vessel_scoring.models import train_model_on_data
from vessel_scoring.evaluate_model import evaluate_model, compare_models
from IPython.core.display import display, HTML, Markdown
import numpy as np
import sys
from sklearn import metrics
from vessel_scoring.logistic_model import LogisticModel
def make_model(seed=4321):
    return LogisticModel(colspec=dict(
        windows=[1800, 3600, 10800, 21600, 43200, 86400],
        measures=['measure_daylight', 'measure_speed']), order=6, random_state=seed)
def load_data(seed=4321):
    # Data supplied by Kristina
    _, train_lline,  valid_lline, test_lline = data.load_dataset_by_vessel(
            '../datasets/kristina_longliner.measures.npz', seed)
    _, train_trawl,  valid_trawl, test_trawl = data.load_dataset_by_vessel(
            '../datasets/kristina_trawl.measures.npz', seed)
    _, train_pseine, valid_pseine, test_pseine = data.load_dataset_by_vessel(
            '../datasets/kristina_ps.measures.npz', seed)
    # Slow transits (used to train models to avoid classifying slow transits as fishing)
    TRANSIT_WEIGHT = 10
    x_tran, xtrain_tran, xcross_tran, xtest_tran = data.load_dataset_by_vessel(
                                        '../datasets/slow-transits.measures.npz', even_split=False, seed=seed)
    xtrain_tran = utils.clone_subset(xtrain_tran, test_lline.dtype)
    xcross_tran = utils.clone_subset(xcross_tran, test_lline.dtype)
    xtest_tran = utils.clone_subset(xtest_tran, test_lline.dtype)
    train_tran = np.concatenate([xtrain_tran, xcross_tran] * TRANSIT_WEIGHT)
    train = {'longliner': np.concatenate([train_lline, valid_lline, train_tran]), 
                'trawler': np.concatenate([train_trawl, valid_trawl, train_tran]),
                'purse_seine': np.concatenate([train_pseine, valid_pseine, train_tran])}
    
    test = {'longliner': test_lline, 
            'trawler': test_trawl, 
            'purse_seine': test_pseine}
    
    return train, test
def get_seeds(count):
    np.random.seed(4321)
    return np.random.randint(4294967295, size=count)
Explanation: Examine sensitivity of current best model (Gear-Specific, Multi-Window 
Logistic Model with is-daylight) with respect to the various random
seeds that are used during training.
Random seed for model itself. Expect minimal effect
Random seed for splitting training / testing data. Perhaps larger effect
As it turns out (see below), neither makes much of a difference. Since this
test was originally convceived the MMSI selection became a lot less random
in order to better match the number of test and training MMSI. As a result,
this lack of sensitivity may not hold as we add new MMSI. That said, we can
expect the results to improve as we add more MMSI going forward.
End of explanation
N_SEEDS = 10
train_data, test_data = load_data()
for gear in ['purse_seine', 'trawler', 'longliner']:
    X_test = test_data[gear]
    display(HTML("<h2>{}</h2>".format(gear.replace('_', ' ').title())))
        
    predictions = []
    trained_models = []
    for seed in get_seeds(N_SEEDS):
        mdl = make_model(seed)
        
        train_model_on_data(mdl, train_data[gear])
        
        trained_models.append((seed, mdl))
        
        predictions.append((seed, (mdl.predict_proba(X_test)[:,1] > 0.5), X_test['classification'] > 0.5))
    lines = ["|Model|Recall|Precision|F1-Score|",
         "|-----|------|---------|--------|"]
    for name, pred, actual in predictions:
        lines.append("|{}|{:.2f}|{:.2f}|{:.2f}|".format(name, 
                                            metrics.recall_score(actual, pred),
                                            metrics.precision_score(actual, pred), 
                                            metrics.f1_score(actual, pred)))
    display(Markdown('\n'.join(lines)))
    compare_models(trained_models, X_test)
    
    display(HTML("<hr/>"))
Explanation: First investigate sensitivity of the LogisiticModels to different seeds
End of explanation
for gear in ['purse_seine', 'trawler', 'longliner']:
    X_test = test_data[gear]
    display(HTML("<h2>{}</h2>".format(gear.replace('_', ' ').title())))
        
    predictions = []
    trained_models = []
    for seed in get_seeds(N_SEEDS):
        mdl = make_model()
        
        train_data, test_data = load_data(seed)
        
        train_model_on_data(mdl, train_data[gear])
        
        trained_models.append((seed, mdl))
        
        predictions.append((seed, (mdl.predict_proba(X_test)[:,1] > 0.5), X_test['classification'] > 0.5))
    lines = ["|Model|Recall|Precision|F1-Score|",
         "|-----|------|---------|--------|"]
    for name, pred, actual in predictions:
        lines.append("|{}|{:.2f}|{:.2f}|{:.2f}|".format(name, 
                                            metrics.recall_score(actual, pred),
                                            metrics.precision_score(actual, pred), 
                                            metrics.f1_score(actual, pred)))
    display(Markdown('\n'.join(lines)))
    compare_models(trained_models, X_test)
    
    display(HTML("<hr/>"))
Explanation: Essentially no difference when setting the seed for different runs.
What about setting the seed when loading the data?
End of explanation
<END_TASK>