<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Scikit-learn from CSV
This notebook reads the CSV data written out by the Dataflow program of 1_explore.ipynb and trains a scikit-learn model on Cloud ML Engine.
Step1:  Train sklearn model locally
Step2:  Training sklearn model on CMLE
<ASSISTANT_TASK:>
Python Code:
%pip install cloudml-hypertune
BUCKET = 'cloud-training-demos-ml'
PROJECT = 'cloud-training-demos'
REGION = 'us-central1'
import os
os.environ['BUCKET'] = BUCKET
os.environ['PROJECT'] = PROJECT
os.environ['REGION'] = REGION
%%bash
gcloud config set project $PROJECT
gcloud config set compute/region $REGION
%load_ext autoreload
%aimport ltgpred
Explanation: Scikit-learn from CSV
This notebook reads the CSV data written out by the Dataflow program of 1_explore.ipynb and trains a scikit-learn model on Cloud ML Engine.
End of explanation
!mkdir -p preproc/csv
!gsutil cp gs://$BUCKET/lightning/preproc_0.02_32_2/csv/*-00000-* preproc/csv
import pandas as pd
df = pd.read_csv('preproc/csv/train-00000-of-00522', 
        header=None,
        names=[
            'cx', 'cy', 'lat', 'lon', 'mean_ref_sm', 'max_ref_sm',
            'mean_ref_big', 'max_ref_big', 'ltg_sm', 'ltg_big', 'has_ltg'
        ])
del df['has_ltg']
#df = pd.get_dummies(df)
df.head()
%%bash
export CLOUDSDK_PYTHON=$(which python3)
OUTDIR=skl_trained
DATADIR=${PWD}/preproc/csv
rm -rf $OUTDIR
gcloud ml-engine local train \
    --module-name=trainer.train_skl --package-path=${PWD}/ltgpred/trainer \
    -- \
    --job-dir=$OUTDIR --train_data=${DATADIR}/train* --eval_data=${DATADIR}/eval*
Explanation: Train sklearn model locally
End of explanation
%%writefile largemachine.yaml
trainingInput:
  scaleTier: CUSTOM
  masterType: complex_model_l
%%bash
export CLOUDSDK_PYTHON=$(which python3)
OUTDIR=gs://${BUCKET}/lightning/skl_trained
DATADIR=gs://$BUCKET/lightning/preproc_0.02_32_2/csv
JOBNAME=ltgpred_skl_$(date -u +%y%m%d_%H%M%S)
gsutil -m rm -rf $OUTDIR
gcloud ml-engine jobs submit training $JOBNAME \
    --module-name=ltgpred.trainer.train_skl --package-path=${PWD}/ltgpred --job-dir=$OUTDIR \
    --region=${REGION} --scale-tier=custom --config=largemachine.yaml \
    --python-version=3.5 --runtime-version=1.9 \
    -- \
    --train_data=${DATADIR}/train-001* --eval_data=${DATADIR}/eval-0000*
Explanation: Training sklearn model on CMLE
End of explanation
<END_TASK>