<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Digit Recognizer Notebook
In this Kaggle competition 
MNIST ("Modified National Institute of Standards and Technology") is the de facto “hello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.
In this competition, your goal is to correctly identify digits from a dataset of tens of thousands of handwritten images.
Install necessary packages
We can install the necessary package by either running pip install --user &lt;package_name&gt; or include everything in a requirements.txt file and run pip install --user -r requirements.txt. We have put the dependencies in a requirements.txt file so we will use the former method.
Step1:  Imports
In this section we import the packages we need for this example. Make it a habit to gather your imports in a single place. It will make your life easier if you are going to transform this notebook into a Kubeflow pipeline using Kale.
Step2:  Set random seed for reproducibility and ignore warning messages.
Step3:  Download data
Step4:  Load and preprocess data
In this section, we load z the dataset to get it in a ready-to-use form by the model. First, let us load and analyze the data.
Load data
The data are in csv format, thus, we use the handy read_csv pandas method. There is one train data set and two test sets (one public and one private).
Step5:  Let us now explore the data
To this end, we use the pandas head method to visualize the 1st five rows of our data set.
Step6:  Data dimension
lets check train and test dimensions
Step7:  Preprocess data
We are now ready to transform the data set and split the dataset into features and the target variables.
Step8:  Define and train the model
we define models with convoolution and dropout layers in our model architecture
Step9:  We are now ready to compile and fit the model.
Step10:  Evaluate the model
Evaluate the model and print the results
Step11:  Confusion matrix
Step12:  Submission
Last but note least, we create our submission to the Kaggle competition. The submission is just a csv file with the specified columns.
<ASSISTANT_TASK:>
Python Code:
!pip install -r requirements.txt --quiet
Explanation: Digit Recognizer Notebook
In this Kaggle competition 
MNIST ("Modified National Institute of Standards and Technology") is the de facto “hello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. As new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.
In this competition, your goal is to correctly identify digits from a dataset of tens of thousands of handwritten images.
Install necessary packages
We can install the necessary package by either running pip install --user &lt;package_name&gt; or include everything in a requirements.txt file and run pip install --user -r requirements.txt. We have put the dependencies in a requirements.txt file so we will use the former method.
End of explanation
import os
import datetime
import numpy as np
import pandas as pd
import pickle
import zipfile
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
import tensorflow as tf
from tensorflow import keras, optimizers
from tensorflow.keras.metrics import SparseCategoricalAccuracy
from tensorflow.keras.losses import SparseCategoricalCrossentropy
from tensorflow.keras import layers
print("tensorflow version: ", tf.__version__)
# hyper parameters
LR = 1e-3
EPOCHS = 2
BATCH_SIZE = 64
CONV_DIM1 = 56
CONV_DIM2 = 100
Explanation: Imports
In this section we import the packages we need for this example. Make it a habit to gather your imports in a single place. It will make your life easier if you are going to transform this notebook into a Kubeflow pipeline using Kale.
End of explanation
tf.random.set_seed(42)
np.random.seed(42)
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
# Setting the graph style
plt.rc('figure', autolayout=True)
plt.rc('axes', titleweight='bold', 
        titlesize=15)
plt.rc('font', size=12)
Explanation: Set random seed for reproducibility and ignore warning messages.
End of explanation
import zipfile
import wget
import os
# download files
data_path = os.getcwd()
# data link
train_link = 'https://github.com/josepholaide/examples/blob/master/digit-recognition-kaggle-competition/data/train.csv.zip?raw=true'
test_link = 'https://github.com/josepholaide/examples/blob/master/digit-recognition-kaggle-competition/data/test.csv.zip?raw=true'
sample_submission = 'https://raw.githubusercontent.com/josepholaide/examples/master/digit-recognition-kaggle-competition/data/sample_submission.csv'
# download data
wget.download(train_link, f'{data_path}/train_csv.zip')
wget.download(test_link, f'{data_path}/test_csv.zip')
wget.download(sample_submission, f'{data_path}/sample_submission.csv')
with zipfile.ZipFile(f"{data_path}/train_csv.zip","r") as zip_ref:
    zip_ref.extractall(data_path)
with zipfile.ZipFile(f"{data_path}/test_csv.zip","r") as zip_ref:
    zip_ref.extractall(data_path)
Explanation: Download data
End of explanation
data_path = os.getcwd()
# Data Path
train_data_path = data_path + '/train.csv'
test_data_path = data_path + '/test.csv'
sample_submission_path = data_path + '/sample_submission.csv'
# Loading dataset into pandas 
train_df = pd.read_csv(train_data_path)
test_df = pd.read_csv(test_data_path)
ss = pd.read_csv(sample_submission_path)
Explanation: Load and preprocess data
In this section, we load z the dataset to get it in a ready-to-use form by the model. First, let us load and analyze the data.
Load data
The data are in csv format, thus, we use the handy read_csv pandas method. There is one train data set and two test sets (one public and one private).
End of explanation
train_df.head()
Explanation: Let us now explore the data
To this end, we use the pandas head method to visualize the 1st five rows of our data set.
End of explanation
train_df.shape, test_df.shape
# join train and test together
ntrain = train_df.shape[0]
ntest = test_df.shape[0]
all_data = pd.concat((train_df, test_df)).reset_index(drop=True)
print("all_data size is : {}".format(all_data.shape))
Explanation: Data dimension
lets check train and test dimensions
End of explanation
all_data_X = all_data.drop('label', axis=1)
all_data_y = all_data.label
# Reshape image in 3 dimensions (height = 28px, width = 28px , channel = 1)
all_data_X = all_data_X.values.reshape(-1,28,28,1)
# Normalize the data
all_data_X = all_data_X / 255.0
#Get the new dataset
X = all_data_X[:ntrain].copy()
y = all_data_y[:ntrain].copy()
# split the dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=42)
Explanation: Preprocess data
We are now ready to transform the data set and split the dataset into features and the target variables.
End of explanation
def build_model(hidden_dim1=int(CONV_DIM1), hidden_dim2=int(CONV_DIM2), DROPOUT=0.5):
    model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(filters = hidden_dim1, kernel_size = (5,5),padding = 'Same', 
                 activation ='relu'),
    tf.keras.layers.Dropout(DROPOUT),
    tf.keras.layers.Conv2D(filters = hidden_dim2, kernel_size = (3,3),padding = 'Same', 
                 activation ='relu'),
    tf.keras.layers.Dropout(DROPOUT),
    tf.keras.layers.Conv2D(filters = hidden_dim2, kernel_size = (3,3),padding = 'Same', 
                 activation ='relu'),
    tf.keras.layers.Dropout(DROPOUT),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(10, activation = "softmax")
    ])
    model.build(input_shape=(None,28,28,1))
    
    return model
model = build_model()
# display the model summary
model.summary()
Explanation: Define and train the model
we define models with convoolution and dropout layers in our model architecture
End of explanation
model.compile(optimizers.Adam(learning_rate=float(LR)), 
              loss=SparseCategoricalCrossentropy(), 
              metrics=SparseCategoricalAccuracy(name='accuracy'))
history = model.fit(np.array(X_train), np.array(y_train), 
                    validation_split=.1, batch_size=int(BATCH_SIZE), epochs=int(EPOCHS))
Explanation: We are now ready to compile and fit the model.
End of explanation
# Evaluate the model and print the results
test_loss, test_acc = model.evaluate(np.array(X_test),  np.array(y_test), verbose=0)
print("Test_loss: {}, Test_accuracy: {} ".format(test_loss,test_acc))
Explanation: Evaluate the model
Evaluate the model and print the results
End of explanation
y_pred = np.argmax(model.predict(X_test), axis=-1)
cm = confusion_matrix(y_test, y_pred)
plt.figure(figsize=(7,7))
sns.heatmap(cm, fmt='g', cbar=False, annot=True, cmap='Blues')
plt.title('confusion_matrix')
plt.ylabel('True label')
plt.xlabel('Predicted label')
plt.show()
Explanation: Confusion matrix
End of explanation
test = all_data_X[ntrain:].copy()
submission_file = np.argmax(model.predict(test), axis=-1)
ss['Label'] = submission_file
ss.to_csv('sub.csv', index=False)
ss.head()
Explanation: Submission
Last but note least, we create our submission to the Kaggle competition. The submission is just a csv file with the specified columns.
End of explanation
<END_TASK>