<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 lesson1
Step1:  Finetuning and Training
Step2:  Use a pretrained VGG model with our Vgg16 class
Step4:  The original pre-trained Vgg16 class classifies images into one of the 1000 categories. This number of categories depends on the dataset which Vgg16 was trained with. (http
Step5:  Generate Predictions
Step6:  Validate Predictions
Calculate predictions on validation set, so we can find correct and incorrect examples
Step7:  (TODO) look at data to improve model
confusion matrix
Step8:  Submit Predictions to Kaggle!
This section also depends on which dataset you use (and which Kaggle competition you are participating)
<ASSISTANT_TASK:>
Python Code:
# make some Python3 functions available on Python2
from __future__ import division, print_function
import sys
print(sys.version_info)
import theano
print(theano.__version__)
import keras
print(keras.__version__)
# FloydHub: check data
%ls /input/dogscats/
# check current directory
%pwd
%ls
# see some files are loaded fine
%cat floyd_requirements.txt
# check no Keras2 specific function is used (when Keras1 is used)
%cat utils.py
#Create references to important directories we will use over and over
import os, sys
current_dir = os.getcwd()
LESSON_HOME_DIR = current_dir
# FloydHub
DATA_HOME_DIR = "/input/dogscats/"
OUTPUT_HOME_DIR = "/output/"
# alternatively, for local
#DATA_HOME_DIR = current_dir+'/data/redux'
#import modules
from utils import *
from vgg16 import Vgg16
#Instantiate plotting tool
#In Jupyter notebooks, you will need to run this command before doing any plotting
%matplotlib inline
Explanation: lesson1: Convolutional Neural Networks with dogscats
Let's classify images using deep learning and submit the result to Kaggle!
Prerequisite
This notebook assumes Keras with Theano backend.
- TODO: make TensorFlow version as another notebook
It also assumes that you will run it on either one of these two cases:
- Floydhub (--env theano:py2 -> Theano rel-0.8.2 + Keras 1.2.2 on Python2)
- local conda virtual environment (Theano 0.9.0 + Keras 2.0.4 on Python3)
Refer to this FloydHub document for available FloydHub environments.
Setup
Make sure to have these files in the parent directory of the directory where you execute this notebook.
available in the official repo for Keras1 on Python2 (rename from original files)
utils_keras1.py
vgg16_keras1.py
vgg16bn_keras1.py
available in the unofficial repo for Keras2 on Python3
utils.py
vgg16.py
vgg16bn.py
The directory structure looks like this. Please modifiy the symlinks according to your environment.
(*) only for FloydHub
(**) only for local
floyd_requirements.txt (*)
floydhub.data.unzip/   (*)
floydhub.data.zipped/  (*)
    dogscats.zip
lesson1/
    data/ (**)
        redux/
            train/
                cat.437.jpg
                dog.9924.jpg
                ...
            test/
                231.jpg
                325.jpg
                ...
    dogscats_run.ipynb
    floyd_requirements.txt -&gt; ../floyd_requirements.txt (*)
    utils.py -&gt; ../utils(_keras1).py
    vgg16.py -&gt; ../vgg16(_keras1).py
    vgg16bn.py -&gt; ../vgg16bn(_keras1).py
utils.py
utils_keras1.py
vgg16.py
vgg16_keras1.py
vgg16bn.py
vgg16bn_keras1.py
Prepare data
The details of data preparation largely depends on which dataset you use. In this section, we will use a pre-organized dataset from http://files.fast.ai/files/dogscats.zip
For another example of data preparation, please refer to this notebook
How the dataset looks like
After extracting the dogscats.zip file, the directory structure look like this.
dogscats/
    models/
    sample/
        train/
            cats/
                cat.394.jpg
                ... (8 items)
            dogs/
                dog.1402.jpg
                ... (8 items)
        valid/
            cats/
                cat.10435.jpg
                ... (4 items)
            dogs/
                dog.10459.jpg
                ... (4 items)
            features.npy
            labels.npy
    test1/
        1.jpg
        10.jpg
        100.jpg
        ... (12500 items)
    train/
        cats/
            cat.0.jpg
            cat.1.jpg
            cat.3.jpg
            ... (11500 items)
        dogs/
            cat.0.jpg
            cat.1.jpg
            cat.2.jpg
            cat.4.jpg
            ... (11500 items)
    valid/
        cats/
            cat.2.jpg
            cat.5.jpg
            ... (1000 item. these are copied from train/cats/ directory)
        dogs/
            dog.3.jpg
            dog.9.jpg
            ... (1000 item. these are copied from train/dogs/ directory)
FloydHub
The cell below shows how to update data to FloydHub.
```
from the directory which this notebook is executed
cd ../floydhub.data.zipped/; pwd
expected: empty
ls -l
wget http://files.fast.ai/files/dogscats.zip
upload the zipped dataset to floydnet, and create a floydnet dataset
floyd data init dogscats.zipped
floyd data upload
```
Using the data we have just uploaded to FloydHub, let's unzip it on FloydHub.
```
from the directory which this notebook is executed
cd ../floydhub.fast.ai.data.unzip/; pwd
expected: empty
ls -l
floyd init dogscats.unzip
floyd run --gpu --data [data ID of the uploaded zip] "unzip /input/dogscats.zip -d /output"
```
Please note:
- the data ID should be the one you see from the above step
- the mounted data is available in /input/ directory, and you need to direct the unzipped files to /output/ directory
local
TODO
Run the notebook
Now let's run the notebook in the environment of your choice.
```
from the directory which this notebook is executed
cd ./; pwd
FloydHub
floyd init dogscats
floyd run --mode jupyter --data [data ID of unzipped data] --env theano:py2 --gpu
alternatively, for local
jupyter notebook
```
and check ~/.keras/keras.json
```
mkdir ~/.keras
FloydHub (Keras1)
echo '{
    "image_dim_ordering": "th",
    "epsilon": 1e-07,
    "floatx": "float32",
    "backend": "theano"
}' > ~/.keras/keras.json
alternatively, for local (Keras2)
echo '{
"image_data_format": "channels_first",
"backend": "theano",
"floatx": "float32",
"epsilon": 1e-07
}' > ~/.keras/keras.json
```
Finally, let's start running the notebook.
End of explanation
%cd $DATA_HOME_DIR
#Set path to sample/ path if desired
path = DATA_HOME_DIR + '/' #'/sample/'
test_path = DATA_HOME_DIR + '/test1/' #We use all the test data
# FloydHub
# data needs to be output under /output
# if results_path cannot be created, execute mkdir directly in the terminal
results_path = OUTPUT_HOME_DIR + '/results/'
%mkdir results_path
train_path = path + '/train/'
valid_path = path + '/valid/'
Explanation: Finetuning and Training
End of explanation
# As large as you can, but no larger than 64 is recommended.
#batch_size = 8
batch_size = 64
no_of_epochs=3
Explanation: Use a pretrained VGG model with our Vgg16 class
End of explanation
vgg = Vgg16()
# Grab a few images at a time for training and validation.
batches = vgg.get_batches(train_path, batch_size=batch_size)
val_batches = vgg.get_batches(valid_path, batch_size=batch_size*2)
# Finetune: note that the vgg model is compiled inside the finetune method.
vgg.finetune(batches)
# Fit: note that we are passing in the validation dataset to the fit() method
# For each epoch we test our model against the validation set
latest_weights_filename = None
# FloydHub (Keras1)
for epoch in range(no_of_epochs):
    print("Running epoch: %d" % epoch)
    vgg.fit(batches, val_batches, nb_epoch=1)
    latest_weights_filename = 'ft%d.h5' % epoch
    vgg.model.save_weights(results_path+latest_weights_filename)
print("Completed %s fit operations" % no_of_epochs)
# alternatively, for local (Keras2)
for epoch in range(no_of_epochs):
    print("Running epoch: %d" % epoch)
    vgg.fit(batches, val_batches, batch_size, nb_epoch=1)
    latest_weights_filename = 'ft%d.h5' % epoch
    vgg.model.save_weights(results_path+latest_weights_filename)
print("Completed %s fit operations" % no_of_epochs)
Explanation: The original pre-trained Vgg16 class classifies images into one of the 1000 categories. This number of categories depends on the dataset which Vgg16 was trained with. (http://image-net.org/challenges/LSVRC/2014/browse-synsets)
In order to classify images into the categories which we prepare (2 categories of dogs/cats, in this notebook), fine-tuning technology is useful. It:
- keeps the most weights from the pre-trained Vgg16 model, but modifies only a few parts of the weights
- changes the dimension of the output layer (from 1000 to 2, in this notebook)
End of explanation
# OUTPUT_HOME_DIR, not DATA_HOME_DIR due to FloydHub restriction
%cd $OUTPUT_HOME_DIR
%mkdir -p test1/unknown
%cd $OUTPUT_HOME_DIR/test1
%cp $test_path/*.jpg unknown/
# rewrite test_path
test_path = OUTPUT_HOME_DIR + '/test1/' #We use all the test data
batches, preds = vgg.test(test_path, batch_size = batch_size*2)
print(preds[:5])
filenames = batches.filenames
print(filenames[:5])
# You can verify the column ordering by viewing some images
from PIL import Image
Image.open(test_path + filenames[2])
#Save our test results arrays so we can use them again later
save_array(results_path + 'test_preds.dat', preds)
save_array(results_path + 'filenames.dat', filenames)
Explanation: Generate Predictions
End of explanation
vgg.model.load_weights(results_path+latest_weights_filename)
val_batches, probs = vgg.test(valid_path, batch_size = batch_size)
filenames = val_batches.filenames
expected_labels = val_batches.classes #0 or 1
#Round our predictions to 0/1 to generate labels
our_predictions = probs[:,0]
our_labels = np.round(1-our_predictions)
Explanation: Validate Predictions
Calculate predictions on validation set, so we can find correct and incorrect examples:
End of explanation
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(expected_labels, our_labels)
plot_confusion_matrix(cm, val_batches.class_indices)
Explanation: (TODO) look at data to improve model
confusion matrix
End of explanation
#Load our test predictions from file
preds = load_array(results_path + 'test_preds.dat')
filenames = load_array(results_path + 'filenames.dat')
#Grab the dog prediction column
isdog = preds[:,1]
print("Raw Predictions: " + str(isdog[:5]))
print("Mid Predictions: " + str(isdog[(isdog < .6) & (isdog > .4)]))
print("Edge Predictions: " + str(isdog[(isdog == 1) | (isdog == 0)]))
# sneaky trick to round down our edge predictions
# Swap all ones with .95 and all zeros with .05
isdog = isdog.clip(min=0.05, max=0.95)
#Extract imageIds from the filenames in our test/unknown directory 
filenames = batches.filenames
ids = np.array([int(f[8:f.find('.')]) for f in filenames])
subm = np.stack([ids,isdog], axis=1)
subm[:5]
# FloydHub
%cd $OUTPUT_HOME_DIR
# alternatively, for local
#%cd $DATA_HOME_DIR
submission_file_name = 'submission1.csv'
np.savetxt(submission_file_name, subm, fmt='%d,%.5f', header='id,label', comments='')
from IPython.display import FileLink
# FloydHub
%cd $OUTPUT_HOME_DIR
FileLink(submission_file_name)
# alternatively, for local
#%cd $LESSON_HOME_DIR
#FileLink('data/redux/'+submission_file_name)
Explanation: Submit Predictions to Kaggle!
This section also depends on which dataset you use (and which Kaggle competition you are participating)
End of explanation
<END_TASK>