<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Load MNIST Data
Step1:  Data Agumentation
image rotation
image width shift
Step2:  Train Parameter
Step3:  Model 1 Architecture
Convolution + Convolution + MaxPool + Dropout
Convolution + Convolution + MaxPool + Dropout
Convolution + MaxPool + Dropout
Dense + Dropout
Dense + Dropout
Output
Step4:  Model 2 Architecture
Convolution * 2 + MaxPool + Dropout
Convolution * 2 + MaxPool + Dropout
Convolution + MaxPool + Dropout
Dense + Dropout
Dense + Dropout
Output
Step5:  Model 3 Architecture
Convolution + Convolution + MaxPool + Dropout
Convolution + Convolution + MaxPool + Dropout
Convolution + MaxPool + Dropout
Dense + Dropout
Dense + Dropout
Output
Step6:  Evaluate
<ASSISTANT_TASK:>
Python Code:
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
# test data
test_images = mnist.test.images.reshape(10000, 28, 28, 1)
test_labels = mnist.test.labels[:]
Explanation: Load MNIST Data
End of explanation
augmentation_size = 110000
images = np.concatenate((mnist.train.images.reshape(55000, 28, 28, 1), mnist.validation.images.reshape(5000, 28, 28, 1)), axis=0)
labels = np.concatenate((mnist.train.labels, mnist.validation.labels), axis=0)
datagen_list = [
                ImageDataGenerator(rotation_range=20),
                ImageDataGenerator(rotation_range=30),
                ImageDataGenerator(width_shift_range=0.1),
                ImageDataGenerator(width_shift_range=0.2),
               ]
for datagen in datagen_list:
    datagen.fit(images)
    for image, label in datagen.flow(images, labels, batch_size=augmentation_size, shuffle=False):
        images = np.concatenate((images, image), axis=0)
        labels = np.concatenate((labels, label), axis=0)
        break
print(images.shape)
print(labels.shape)
Explanation: Data Agumentation
image rotation
image width shift
End of explanation
model_1_filter_size = 3
model_2_filter_size = 5
model_3_filter_size = 7
epochs = 10
Explanation: Train Parameter
End of explanation
model1 = Sequential([Convolution2D(filters=64, kernel_size=(model_1_filter_size, model_1_filter_size), padding='same', activation='elu', input_shape=(28, 28, 1)), 
                     Convolution2D(filters=128, kernel_size=(model_1_filter_size, model_1_filter_size), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Convolution2D(filters=128, kernel_size=(model_1_filter_size, model_1_filter_size), padding='same', activation='elu'),
                     Convolution2D(filters=128, kernel_size=(model_1_filter_size, model_1_filter_size), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Convolution2D(filters=128, kernel_size=(model_1_filter_size, model_1_filter_size), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Flatten(),
                     Dense(1024, activation='elu'),
                     Dropout(0.5),
                     Dense(1024, activation='elu'),
                     Dropout(0.5),
                     Dense(10, activation='softmax'),
                     ])
model1.compile(optimizer=Adam(lr=0.0005), loss='categorical_crossentropy', metrics=['accuracy'])
model1.fit(images, labels, batch_size=256, epochs=epochs, shuffle=True, verbose=1, validation_data=(test_images, test_labels))
model_json = model1.to_json()
with open("model1.json", "w") as json_file:
    json_file.write(model_json)
model1.save_weights("model1.h5")
print("Saved model to disk")
Explanation: Model 1 Architecture
Convolution + Convolution + MaxPool + Dropout
Convolution + Convolution + MaxPool + Dropout
Convolution + MaxPool + Dropout
Dense + Dropout
Dense + Dropout
Output
End of explanation
model2 = Sequential([Convolution2D(filters=64, kernel_size=(model_2_filter_size, model_2_filter_size), padding='same', activation='elu', input_shape=(28, 28, 1)), 
                     Convolution2D(filters=128, kernel_size=(model_2_filter_size, model_2_filter_size), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Convolution2D(filters=128, kernel_size=(model_2_filter_size, model_2_filter_size), padding='same', activation='elu'),
                     Convolution2D(filters=128, kernel_size=(model_2_filter_size, model_2_filter_size), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Convolution2D(filters=128, kernel_size=(model_2_filter_size, model_2_filter_size), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Flatten(),
                     Dense(1024, activation='elu'),
                     Dropout(0.5),
                     Dense(1024, activation='elu'),
                     Dropout(0.5),
                     Dense(10, activation='softmax'),
                     ])
model2.compile(optimizer=Adam(lr=0.0005), loss='categorical_crossentropy', metrics=['accuracy'])
model2.fit(images, labels, batch_size=256, epochs=epochs, shuffle=True, verbose=1, validation_data=(test_images, test_labels))
model_json = model2.to_json()
with open("model2.json", "w") as json_file:
    json_file.write(model_json)
model2.save_weights("model2.h5")
print("Saved model to disk")
Explanation: Model 2 Architecture
Convolution * 2 + MaxPool + Dropout
Convolution * 2 + MaxPool + Dropout
Convolution + MaxPool + Dropout
Dense + Dropout
Dense + Dropout
Output
End of explanation
model3 = Sequential([Convolution2D(filters=64, kernel_size=(model_3_filter_size, model_3_filter_size), padding='same', activation='elu', input_shape=(28, 28, 1)), 
                     Convolution2D(filters=128, kernel_size=(model_3_filter_size, model_3_filter_size), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Convolution2D(filters=128, kernel_size=(model_3_filter_size, model_3_filter_size), padding='same', activation='elu'),
                     Convolution2D(filters=128, kernel_size=(model_3_filter_size, model_3_filter_size), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Convolution2D(filters=128, kernel_size=(model_3_filter_size, model_3_filter_size), padding='same', activation='elu'),
                     MaxPooling2D(pool_size=(2, 2)),
                     Dropout(0.5),
                     Flatten(),
                     Dense(1024, activation='elu'),
                     Dropout(0.5),
                     Dense(1024, activation='elu'),
                     Dropout(0.5),
                     Dense(10, activation='softmax'),
                     ])
model3.compile(optimizer=Adam(lr=0.0005), loss='categorical_crossentropy', metrics=['accuracy'])
model3.fit(images, labels, batch_size=256, epochs=epochs, shuffle=True, verbose=1, validation_data=(test_images, test_labels))
model_json = model3.to_json()
with open("model3.json", "w") as json_file:
    json_file.write(model_json)
model3.save_weights("model3.h5")
print("Saved model to disk")
Explanation: Model 3 Architecture
Convolution + Convolution + MaxPool + Dropout
Convolution + Convolution + MaxPool + Dropout
Convolution + MaxPool + Dropout
Dense + Dropout
Dense + Dropout
Output
End of explanation
# load json and create model
def model_open(name, test_images, test_labels):
    json_file = open(name + '.json', 'r')
    loaded_model_json = json_file.read()
    json_file.close()
    loaded_model = model_from_json(loaded_model_json)
    # load weights into new model
    loaded_model.load_weights(name + '.h5')
    print("Loaded model from disk")
    loaded_model.compile(optimizer=Adam(lr=0.0005), loss='categorical_crossentropy', metrics=['acc'])
    prob = loaded_model.predict_proba(test_images)
    acc = np.mean(np.equal(np.argmax(prob, axis=1), np.argmax(test_labels, axis=1)))
    print('\nmodel : %s, test accuracy : %4f\n' %(name, acc))
    return prob
prob_1 = model_open('model1', test_images, test_labels)
prob_2 = model_open('model2', test_images, test_labels)
prob_3 = model_open('model3', test_images, test_labels)
final_prob = prob_1 * 1 + prob_2 * 2  + prob_3 * 1
final_score = np.mean(np.equal(np.argmax(final_prob, axis=1), np.argmax(test_labels, axis=1)))
print('final test accuracy : ', final_score)
Explanation: Evaluate
End of explanation
<END_TASK>