<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Generate Spark ML Decision Tree
Step 0
Step1:  Step 2
Step2:  Step 3
Step3:  Deployment Option 1
Step4:  Model Server Dashboard
Fill in <your-ip> below, then copy/paste to your browser
http
Step5:  TODO
<ASSISTANT_TASK:>
Python Code:
# You may need to Reconnect (more than Restart) the Kernel to pick up changes to these sett
import os
master = '--master spark://127.0.0.1:47077'
conf = '--conf spark.cores.max=1 --conf spark.executor.memory=512m'
packages = '--packages com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.1'
jars = '--jars /root/lib/jpmml-sparkml-package-1.0-SNAPSHOT.jar'
py_files = '--py-files /root/lib/jpmml.py'
os.environ['PYSPARK_SUBMIT_ARGS'] = master \
  + ' ' + conf \
  + ' ' + packages \
  + ' ' + jars \
  + ' ' + py_files \
  + ' ' + 'pyspark-shell'
print(os.environ['PYSPARK_SUBMIT_ARGS'])
from pyspark.ml import Pipeline
from pyspark.ml.feature import RFormula
from pyspark.ml.classification import DecisionTreeClassifier
from pyspark.sql import SparkSession
sparkSession = SparkSession.builder.getOrCreate()
data = sparkSession.read.format("csv") \
  .option("inferSchema", "true").option("header", "true") \
  .load("hdfs://127.0.0.1:39000/datasets/census/census.csv")
data.head()
Explanation: Generate Spark ML Decision Tree
Step 0: Load Libraries, Data, and SparkSession
End of explanation
formula = RFormula(formula = "income ~ .")
classifier = DecisionTreeClassifier()
pipeline = Pipeline(stages = [formula, classifier])
pipelineModel = pipeline.fit(data)
print(pipelineModel)
print(pipelineModel.stages[1].toDebugString)
Explanation: Step 2: Build Spark ML Pipeline with Decision Tree Classifier
End of explanation
from jpmml import toPMMLBytes
pmmlBytes = toPMMLBytes(sparkSession, data, pipelineModel)
print(pmmlBytes.decode("utf-8"))
Explanation: Step 3: Convert Spark ML Pipeline to PMML
End of explanation
from urllib import request
update_url = 'http://<your-ip>:39040/update-pmml/pmml_census'
update_headers = {}
update_headers['Content-type'] = 'application/xml'
req = request.Request(update_url, headers=update_headers, data=pmmlBytes)
resp = request.urlopen(req)
print(resp.status) # Should return Http Status 200 
from urllib import request
evaluate_url = 'http://<your-ip>:39040/evaluate-pmml/pmml_census'
evaluate_headers = {}
evaluate_headers['Content-type'] = 'application/json'
input_params = '{"age":39,"workclass":"State-gov","education":"Bachelors","education_num":13,"marital_status":"Never-married","occupation":"Adm-clerical","relationship":"Not-in-family","race":"White","sex":"Male","capital_gain":2174,"capital_loss":0,"hours_per_week":40,"native_country":"United-States"}' 
encoded_input_params = input_params.encode('utf-8')
req = request.Request(evaluate_url, headers=evaluate_headers, data=encoded_input_params)
resp = request.urlopen(req)
print(resp.read()) # Should return valid classification with probabilities
Explanation: Deployment Option 1:  Mutable Model Deployment
End of explanation
!mkdir -p /root/src/pmml/census/
with open('/root/src/pmml/census/pmml_census.pmml', 'wb') as f:
  f.write(pmmlBytes)
!ls /root/src/pmml/census/pmml_census.pmml
Explanation: Model Server Dashboard
Fill in <your-ip> below, then copy/paste to your browser
http://&lt;your-ip&gt;:47979/hystrix-dashboard/monitor/monitor.html?streams=%5B%7B%22name%22%3A%22%22%2C%22stream%22%3A%22http%3A%2F%2F&lt;your-ip&gt;%3A39043%2Fhystrix.stream%22%2C%22auth%22%3A%22%22%2C%22delay%22%3A%22%22%7D%2C%7B%22name%22%3A%22%22%2C%22stream%22%3A%22http%3A%2F%2F&lt;your-ip&gt;%3A39042%2Fhystrix.stream%22%2C%22auth%22%3A%22%22%2C%22delay%22%3A%22%22%7D%2C%7B%22name%22%3A%22%22%2C%22stream%22%3A%22http%3A%2F%2F&lt;your-ip&gt;%3A39041%2Fhystrix.stream%22%2C%22auth%22%3A%22%22%2C%22delay%22%3A%22%22%7D%2C%7B%22name%22%3A%22%22%2C%22stream%22%3A%22http%3A%2F%2F&lt;your-ip&gt;%3A39040%2Fhystrix.stream%22%2C%22auth%22%3A%22%22%2C%22delay%22%3A%22%22%7D%5D
Deployment Option 2:  Immutable Model Deployment
Save Model to Disk
End of explanation
!start-loadtest.sh $SOURCE_HOME/loadtest/RecommendationServiceStressTest-local-census.jmx
Explanation: TODO:  Trigger Airflow to Build New Docker Image (ie. via Github commit)
Load Test
End of explanation
<END_TASK>