<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 <a href="https
Step1:  Environment Preparation
Install Java 8
Run the cell on the Google Colab to install jdk 1.8.
Note
Step2:  Install BigDL Orca
You can install the latest pre-release version using !pip install --pre --upgrade bigdl-orca.
Step3:  Distributed Keras (v2.3) using Orca APIs
In this guide we will describe how to scale out Keras (v2.3) programs using Orca in 4 simple steps.
Step4:  Step 1
Step5:  This is the only place where you need to specify local or distributed mode. View Orca Context for more details.
Note
Step6:  Step 3
Step7:  Step 4
Step8:  Next, fit and evaluate using the Estimator
Step9:  Finally, evaluate using the Estimator.
Step10:  Now, the accuracy of this model has reached 97%.
<ASSISTANT_TASK:>
Python Code:
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
Explanation: <a href="https://colab.research.google.com/github/intel-analytics/BigDL/blob/branch-2.0/python/orca/colab-notebook/quickstart/keras_lenet_mnist.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
Copyright 2016 The BigDL Authors.
End of explanation
# Install jdk8
!apt-get install openjdk-8-jdk-headless -qq > /dev/null
import os
# Set environment variable JAVA_HOME.
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
!update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
!java -version
Explanation: Environment Preparation
Install Java 8
Run the cell on the Google Colab to install jdk 1.8.
Note: if you run this notebook on your computer, root permission is required when running the cell to install Java 8. (You may ignore this cell if Java 8 has already been set up in your computer).
End of explanation
# Install latest pre-release version of BigDL Orca 
# Installing BigDL Orca from pip will automatically install pyspark, bigdl, and their dependencies.
!pip install --pre --upgrade bigdl-orca
# Install python dependencies
# The tutorial below only supports TensorFlow 1.15
!pip install tensorflow==1.15.0 tensorflow-datasets==2.1.0
Explanation: Install BigDL Orca
You can install the latest pre-release version using !pip install --pre --upgrade bigdl-orca.
End of explanation
# import necesary libraries and modules
import argparse
from bigdl.orca import init_orca_context, stop_orca_context
from bigdl.orca.learn.tf.estimator import Estimator
from bigdl.orca import OrcaContext
Explanation: Distributed Keras (v2.3) using Orca APIs
In this guide we will describe how to scale out Keras (v2.3) programs using Orca in 4 simple steps.
End of explanation
OrcaContext.log_output = True # recommended to set it to True when running BigDL in Jupyter notebook (this will display terminal's  stdout and stderr in the Jupyter notebook).
cluster_mode = "local"
if cluster_mode == "local":  
    init_orca_context(cluster_mode="local", cores=4) # run in local mode
    dataset_dir = "~/tensorflow_datasets"
elif cluster_mode == "k8s":  
    init_orca_context(cluster_mode="k8s", num_nodes=2, cores=2) # run on K8s cluster
elif cluster_mode == "yarn":  
    init_orca_context(cluster_mode="yarn-client", num_nodes=2, cores=2) # run on Hadoop YARN cluster
    dataset_dir = "hdfs:///tensorflow_datasets"
Explanation: Step 1: Init Orca Context
End of explanation
from tensorflow import keras
model = keras.Sequential(
        [keras.layers.Conv2D(20, kernel_size=(5, 5), strides=(1, 1), activation='tanh',
                             input_shape=(28, 28, 1), padding='valid'),
         keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'),
         keras.layers.Conv2D(50, kernel_size=(5, 5), strides=(1, 1), activation='tanh',
                             padding='valid'),
         keras.layers.MaxPooling2D(pool_size=(2, 2), strides=(2, 2), padding='valid'),
         keras.layers.Flatten(),
         keras.layers.Dense(500, activation='tanh'),
         keras.layers.Dense(10, activation='softmax'),
         ]
    )
model.compile(optimizer=keras.optimizers.RMSprop(),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
Explanation: This is the only place where you need to specify local or distributed mode. View Orca Context for more details.
Note: You should export HADOOP_CONF_DIR=/path/to/hadoop/conf/dir when you run on Hadoop YARN cluster. To use tensorflow_datasets on HDFS, you should correctly set HADOOP_HOME, HADOOP_HDFS_HOME, LD_LIBRARY_PATH, etc. For more details, please refer to TensorFlow documentation link.
Step 2: Define the Model
You may define your model, loss and metrics in the same way as in any standard (single node) Keras program.
End of explanation
import tensorflow as tf
import tensorflow_datasets as tfds
def preprocess(data):
  data['image'] = tf.cast(data["image"], tf.float32) / 255.
  return data['image'], data['label']
# get DataSet
mnist_train = tfds.load(name="mnist", split="train", data_dir=dataset_dir)
mnist_test = tfds.load(name="mnist", split="test", data_dir=dataset_dir)
mnist_train = mnist_train.map(preprocess)
mnist_test = mnist_test.map(preprocess)
Explanation: Step 3: Define Train Dataset
You can define the dataset using standard tf.data.Dataset.
End of explanation
from bigdl.orca.learn.tf.estimator import Estimator
est = Estimator.from_keras(keras_model=model)
Explanation: Step 4: Fit with Orca Estimator
First, create an Estimator
End of explanation
max_epoch = 1
est.fit(data=mnist_train,
        batch_size=320,
        epochs=max_epoch,
        validation_data=mnist_test)
Explanation: Next, fit and evaluate using the Estimator
End of explanation
# evaluate and print result
result = est.evaluate(mnist_test)
print(result)
est.save_keras_model("/tmp/mnist_keras.h5")
Explanation: Finally, evaluate using the Estimator.
End of explanation
# stop orca context when program finishes
stop_orca_context()
Explanation: Now, the accuracy of this model has reached 97%.
End of explanation
<END_TASK>