<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 DeepLearning
Introduction
In this notebook, we introduce H2O Deep Learning via fully-connected artificial neural networks. We also show many useful features of H2O such as hyper-parameter optimization, Flow, and checkpointing. There are other notebooks that use more complex convolutional neural networks ranging from LeNet all the way to Inception Resnet V2.
MNIST Dataset
The MNIST database is a well-known academic dataset used to benchmark
classification performance. The data consists of 60,000 training images and
10,000 test images. Each image is a standardized $28^2$ pixel greyscale image of
a single handwritten digit. A sample of the scanned handwritten digits is
shown
Step1:  Specify the response and predictor columns
Step2:  Train Deep Learning model and validate on test set
Step3:  Inspect the model in Flow
It is highly recommended to use Flow to visualize the model training process and to inspect the model before using it for further steps.
Using Crossvalidation
If the value specified for nfolds is a positive integer, N-fold cross-validation is
performed on the training frame and the cross-validation metrics are computed
and stored as model output. 
To disable cross-validation, use nfolds=0, which is the default value.
Advanced users can also specify a fold column that defines the holdout
fold associated with each row. By default, the holdout fold assignment is
random. H2O supports other schemes such as round-robin assignment using the modulo
operator.
Perform 3-fold cross-validation on training_frame
Step4:  Extracting and Handling the Results
We can now extract the parameters of our model, examine the scoring process,
and make predictions on new data.
Step5:  Note
Step6:  Clearly, the model parameters aren't tuned perfectly yet, as 4-5% test set error is rather large.
Step7:  Predicting
Once we have a satisfactory model (as determined by the validation or crossvalidation
metrics), use the h2o.predict() command to compute and store
predictions on new data for additional refinements in the interactive data science
process.
Step8:  Variable Importance
Variable importance allows us to view the absolute and relative predictive strength of
each feature in the prediction task.
Each H2O algorithm class has its own methodology for computing variable importance.
You can enable the variable importance, by setting the variable_importances parameter to True.
H2O’s Deep Learning uses the Gedeon method Gedeon, 1997, which is disabled
by default since it can be slow for large networks. 
If variable importance is a top priority in your analysis, consider training a Distributed Random Forest (DRF) model and compare the generated variable importances.
Step9:  Model Comparison with Grid Search
Grid search provides more subtle insights into the model tuning and selection
process by inspecting and comparing our trained models after the grid search process is complete. 
To learn when and how to select different parameter
configurations in a grid search, refer to Parameters for parameter descriptions
and configurable values.
There are different strategies to explore the hyperparameter combinatorial space
Step10:  Random Grid Search
If the search space is too large you can let the GridSearch algorithm select the parameter, by sampling from the parameter space. 
Just specify how many models (and/or how much training time) you want, and provide a seed to make the random selection deterministic.
Step11:  Model Checkpoints
H2O supporst model checkpoints. You can store the state of training and resume it later.
Checkpointing can be used to reload existing models that were saved to
disk in a previous session. 
To resume model training, use checkpoint model keys (model id) to incrementally
train a specific model using more iterations, more data, different data, and
so forth. To further train the initial model, use it (or its key) as a checkpoint
argument for a new model.
To improve this initial model, start from the previous model and add iterations by
building another model, specifying checkpoint=previous model id, and
changing train samples per iteration, target ratio comm to comp,
or other parameters. Many parameters can be changed between checkpoints,
especially those that affect regularization or performance tuning.
You can use GridSearch with checkpoint restarts to scan a broader range of hyperparameter combinations.
Step12:  Specify a model and a file path. The default path is the current working directory.
Step13:  After restarting H2O, you can load the saved model by specifying the host and model file path. 
Note
Step14:  You can also use the following commands to retrieve a model from its H2O key.
This is useful if you have created an H2O model using the web interface and
want to continue the modeling process in another language, for example R.
<ASSISTANT_TASK:>
Python Code:
import h2o
h2o.init(nthreads=-1)
import os.path
PATH = os.path.expanduser("~/h2o-3/")
test_df = h2o.import_file(PATH + "bigdata/laptop/mnist/test.csv.gz")
train_df = h2o.import_file(PATH + "bigdata/laptop/mnist/train.csv.gz")
Explanation: DeepLearning
Introduction
In this notebook, we introduce H2O Deep Learning via fully-connected artificial neural networks. We also show many useful features of H2O such as hyper-parameter optimization, Flow, and checkpointing. There are other notebooks that use more complex convolutional neural networks ranging from LeNet all the way to Inception Resnet V2.
MNIST Dataset
The MNIST database is a well-known academic dataset used to benchmark
classification performance. The data consists of 60,000 training images and
10,000 test images. Each image is a standardized $28^2$ pixel greyscale image of
a single handwritten digit. A sample of the scanned handwritten digits is
shown
End of explanation
y = "C785"
x = train_df.names[0:784]
train_df[y] = train_df[y].asfactor()
test_df[y] = test_df[y].asfactor()
Explanation: Specify the response and predictor columns
End of explanation
from h2o.estimators.deepwater import H2ODeepWaterEstimator
model = H2ODeepWaterEstimator(
   distribution="multinomial",
   activation="rectifier",
   mini_batch_size=128,
   hidden=[1024,1024],
   hidden_dropout_ratios=[0.5,0.5],      ## for better generalization
   input_dropout_ratio=0.1,
   sparse=True,                          ## can result in speedup for sparse data
   epochs=10)                            ## need more epochs for a better model
model.train(
    x=x, 
    y=y,
    training_frame=train_df,
    validation_frame=test_df
)
model.scoring_history()
model.model_performance(train=True) # training metrics
model.model_performance(valid=True) # validation metrics
Explanation: Train Deep Learning model and validate on test set
End of explanation
model_crossvalidated = H2ODeepWaterEstimator(
   distribution="multinomial",
   activation="rectifier",
   mini_batch_size=128,
   hidden=[1024,1024],
   hidden_dropout_ratios=[0.5,0.5],
   input_dropout_ratio=0.1,
   sparse=True,
   epochs=10,
   nfolds=3
)
model_crossvalidated.train(
    x=x,
    y=y,
    training_frame=train_df
)
Explanation: Inspect the model in Flow
It is highly recommended to use Flow to visualize the model training process and to inspect the model before using it for further steps.
Using Crossvalidation
If the value specified for nfolds is a positive integer, N-fold cross-validation is
performed on the training frame and the cross-validation metrics are computed
and stored as model output. 
To disable cross-validation, use nfolds=0, which is the default value.
Advanced users can also specify a fold column that defines the holdout
fold associated with each row. By default, the holdout fold assignment is
random. H2O supports other schemes such as round-robin assignment using the modulo
operator.
Perform 3-fold cross-validation on training_frame
End of explanation
# View specified parameters of the Deep Learning model
model_crossvalidated.params;
# Examine the trained model
model_crossvalidated
Explanation: Extracting and Handling the Results
We can now extract the parameters of our model, examine the scoring process,
and make predictions on new data.
End of explanation
## Validation error of the original model (using a train/valid split)
model.mean_per_class_error(valid=True)
## Training error of the model trained on 100% of the data
model_crossvalidated.mean_per_class_error(train=True)
## Estimated generalization error of the cross-validated model
model_crossvalidated.mean_per_class_error(xval=True)
Explanation: Note: The validation error is based on the
parameter score_validation_samples, which can be used to sample the validation set (by default, the entire validation set is used).
End of explanation
#ls ../../h2o-docs/src/booklets/v2_2015/source/images/
Explanation: Clearly, the model parameters aren't tuned perfectly yet, as 4-5% test set error is rather large.
End of explanation
predictions = model_crossvalidated.predict(test_df)
predictions.describe()
Explanation: Predicting
Once we have a satisfactory model (as determined by the validation or crossvalidation
metrics), use the h2o.predict() command to compute and store
predictions on new data for additional refinements in the interactive data science
process.
End of explanation
# Train Deep Learning model and validate on test set and save the variable importances
from h2o.estimators.deeplearning import H2ODeepLearningEstimator  ## H2ODeepWaterEstimator doesn't yet have variable importances
model_variable_importances = H2ODeepLearningEstimator(
     distribution="multinomial",
     activation="RectifierWithDropout",  ## shortcut for hidden_dropout_ratios=[0.5,0.5,0.5]
     hidden=[32,32,32],         ## smaller number of neurons to be fast enough on the CPU
     input_dropout_ratio=0.1,
     sparse=True,
     epochs=1,                  ## not interested in a good model here
     variable_importances=True) ## this is not yet implemented for DeepWaterEstimator
model_variable_importances.train(
         x=x,
         y=y,
         training_frame=train_df,
         validation_frame=test_df)
# Retrieve the variable importance
import pandas as pd
pd.DataFrame(model_variable_importances.varimp())
model_variable_importances.varimp_plot(num_of_features=20)
Explanation: Variable Importance
Variable importance allows us to view the absolute and relative predictive strength of
each feature in the prediction task.
Each H2O algorithm class has its own methodology for computing variable importance.
You can enable the variable importance, by setting the variable_importances parameter to True.
H2O’s Deep Learning uses the Gedeon method Gedeon, 1997, which is disabled
by default since it can be slow for large networks. 
If variable importance is a top priority in your analysis, consider training a Distributed Random Forest (DRF) model and compare the generated variable importances.
End of explanation
from h2o.grid.grid_search import H2OGridSearch
hyper_parameters = {
    "hidden":[[200,200,200],[300,300]], 
    "learning_rate":[1e-3,5e-3],
}
model_grid = H2OGridSearch(H2ODeepWaterEstimator, hyper_params=hyper_parameters)
model_grid.train(
    x=x, 
    y=y,
    distribution="multinomial", 
    epochs=50,   ## might stop earlier since we enable early stopping below
    training_frame=train_df, 
    validation_frame=test_df,
    score_interval=2,                ## score no more than every 2 seconds
    score_duty_cycle=0.5,            ## score up to 50% of the time - to enable early stopping
    score_training_samples=1000,     ## use a subset of the training frame for faster scoring
    score_validation_samples=1000,   ## use a subset of the validation frame for faster scoring
    stopping_rounds=3,
    stopping_tolerance=0.05,
    stopping_metric="misclassification",
    sparse = True,
    mini_batch_size=256
)
# print model grid search results
model_grid
for gmodel in model_grid:
    print gmodel.model_id + " mean per class error: " + str(gmodel.mean_per_class_error())
import pandas as pd
grid_results = pd.DataFrame([[m.model_id, m.mean_per_class_error(valid=True)] for m in model_grid])
grid_results
Explanation: Model Comparison with Grid Search
Grid search provides more subtle insights into the model tuning and selection
process by inspecting and comparing our trained models after the grid search process is complete. 
To learn when and how to select different parameter
configurations in a grid search, refer to Parameters for parameter descriptions
and configurable values.
There are different strategies to explore the hyperparameter combinatorial space:
Cartesian Search: test every single combination
Random Search: sample combinations
Cartesian Search
In this example, two different network topologies and two different learning rates are specified. This grid search model trains all 4 different models (all possible combinations of these parameters); other parameter combinations can
be specified for a larger space of models. Note that the models will most likely
converge before the default value of epochs, since early stopping is enabled.
End of explanation
hyper_parameters = {
    "hidden":[[1000,1000],[2000]],
    "learning_rate":[s*1e-3 for s in range(30,100)],
    "momentum_start":[s*1e-3 for s in range(0,900)],
    "momentum_stable":[s*1e-3 for s in range(900,1000)],
}
search_criteria = {"strategy":"RandomDiscrete", "max_models":10, "max_runtime_secs":100, "seed":123456}
model_grid_random_search = H2OGridSearch(H2ODeepWaterEstimator,
    hyper_params=hyper_parameters,
    search_criteria=search_criteria)
model_grid_random_search.train(
    x=x, y=y,
    distribution="multinomial", 
    epochs=50,   ## might stop earlier since we enable early stopping below
    training_frame=train_df, 
    validation_frame=test_df,
    score_interval=2,                ## score no more than every 2 seconds
    score_duty_cycle=0.5,            ## score up to 50% of the wall clock time - scoring is needed for early stopping
    score_training_samples=1000,     ## use a subset of the training frame for faster scoring
    score_validation_samples=1000,   ## use a subset of the validation frame for faster scoring
    stopping_rounds=3,
    stopping_tolerance=0.05,
    stopping_metric="misclassification",
    sparse = True,
    mini_batch_size=256)
grid_results = pd.DataFrame([[m.model_id, m.mean_per_class_error(valid=True)] for m in model_grid_random_search])
grid_results
Explanation: Random Grid Search
If the search space is too large you can let the GridSearch algorithm select the parameter, by sampling from the parameter space. 
Just specify how many models (and/or how much training time) you want, and provide a seed to make the random selection deterministic.
End of explanation
# Re-start the training process on a saved DL model using the ‘checkpoint‘ argument
model_checkpoint = H2ODeepWaterEstimator(
     checkpoint=model.model_id,
     activation="rectifier",
     distribution="multinomial",
     mini_batch_size=128,
     hidden=[1024,1024],
     hidden_dropout_ratios=[0.5,0.5],
     input_dropout_ratio=0.1,
     sparse=True,
     epochs=20)  ## previous model had 10 epochs, so we need to only train for 10 more to get to 20 epochs
model_checkpoint.train(
 x=x,
 y=y,
 training_frame=train_df,
 validation_frame=test_df)
model_checkpoint.scoring_history()
Explanation: Model Checkpoints
H2O supporst model checkpoints. You can store the state of training and resume it later.
Checkpointing can be used to reload existing models that were saved to
disk in a previous session. 
To resume model training, use checkpoint model keys (model id) to incrementally
train a specific model using more iterations, more data, different data, and
so forth. To further train the initial model, use it (or its key) as a checkpoint
argument for a new model.
To improve this initial model, start from the previous model and add iterations by
building another model, specifying checkpoint=previous model id, and
changing train samples per iteration, target ratio comm to comp,
or other parameters. Many parameters can be changed between checkpoints,
especially those that affect regularization or performance tuning.
You can use GridSearch with checkpoint restarts to scan a broader range of hyperparameter combinations.
End of explanation
model_path = h2o.save_model(
     model = model,
     #path = "/tmp/mymodel",
     force = True)
print model_path
!ls -lah $model_path
Explanation: Specify a model and a file path. The default path is the current working directory.
End of explanation
# Load model from disk
saved_model = h2o.load_model(model_path)
Explanation: After restarting H2O, you can load the saved model by specifying the host and model file path. 
Note: The saved model must be the same version used to save the model.
End of explanation
# Retrieve model by H2O key
model = h2o.get_model(model_id=model_checkpoint._id)
model
Explanation: You can also use the following commands to retrieve a model from its H2O key.
This is useful if you have created an H2O model using the web interface and
want to continue the modeling process in another language, for example R.
End of explanation
<END_TASK>