<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 This notebook is going to look at the mistakes made by the chosen model. Specifically, it's going to look at the log loss for each class on the validation set and produce a Hinton diagram for all the classes to show the confusion matrix for this model. Then, it will find the class performing worst and plot example images from this class after processing.
Aiming for it to be written in such a way that the model file chosen can just be swapped in, so we can try it on different models.
Setup and prediction
Doing the exact same thing as in the notebook on Holdout set testing, creating a set of predictions
Step1:  Log loss by class
Creating a bar chart of log loss by class
Step2:  Comparing the score of the network against a uniform benchmark on a scatter plot
Step3:  May be better to take the improvement on uniform as a score
Step4:  Finding out which is the worst
Step5:  What's it being confused with?
Step6:  Worst Datapoints
Now it's probably worth plotting whcih datapoints the network is gettings confused about most, when it's getting confused. So, we find the datapoints with the highest log loss.
Step7:  Checking I've written this right
Step8:  Now we can iterate over all the datapoints and find the worst
Step9:  Now, we want to plot this index along with examples of what it's being confused with. First, find the class it's being confused with and get some indices for that
Step10:  I guess the question is, how is the network supposed to be able to tell the difference in this case? More resolution?
Plotting the above with raw images
Step11:  Yeah, looks like our resizing is losing some of the fine-grained detail. Although, that protist fuzzy olive that was the worst has changed relatively little; it was already fuzzy.
Confusion Matrix
We'd like to plot a confusion matrix for these predictions. To do this, we're going to want to use a Hinton plot, because otherwise it's going to be difficult to interpret the 121 by 121 matrix by eye. First, Scikit-learn will make the confusion matrix array for us
Step12:  Where are the big false positives? We can make a new array just including those, in a configuration we know?
Step13:  So confusion is greatest between classes 10 and 11? We could plot examples of those two classes
<ASSISTANT_TASK:>
Python Code:
# copying imports from holdout notebook...
import pylearn2.utils
import pylearn2.config
import theano
import neukrill_net.dense_dataset
import neukrill_net.utils
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import holoviews as hl
%load_ext holoviews.ipython
import sklearn.metrics
cd ..
settings = neukrill_net.utils.Settings("settings.json")
run_settings = neukrill_net.utils.load_run_settings(
    "run_settings/alexnet_based_norm_global_8aug.json", settings, force=True)
# loading the model
model = pylearn2.utils.serial.load(run_settings['pickle abspath'])
# loading the data
dataset = neukrill_net.dense_dataset.DensePNGDataset(settings_path=run_settings['settings_path'],
                                            run_settings=run_settings['run_settings_path'],
                                                     train_or_predict='train',
                                                     training_set_mode='validation', force=True)
batch_size=500
while dataset.X.shape[0]%batch_size != 0:
    batch_size += 1
n_batches = int(dataset.X.shape[0]/batch_size)
# set this batch size
model.set_batch_size(batch_size)
# compile Theano function
X = model.get_input_space().make_batch_theano()
Y = model.fprop(X)
f = theano.function([X],Y)
%%time
y = np.zeros((dataset.X.shape[0],len(settings.classes)))
for i in xrange(n_batches):
    print("Batch {0} of {1}".format(i+1,n_batches))
    x_arg = dataset.X[i*batch_size:(i+1)*batch_size,:]
    if X.ndim > 2:
        x_arg = dataset.get_topological_view(x_arg)
    y[i*batch_size:(i+1)*batch_size,:] = (f(x_arg.astype(X.dtype).T))
af = run_settings.get("augmentation_factor",1)
if af > 1:
    y_collapsed = np.zeros((int(dataset.X.shape[0]/af), len(settings.classes))) 
    for i,(low,high) in enumerate(zip(range(0,dataset.y.shape[0],af),
                                      range(af,dataset.y.shape[0]+af,af))):
        y_collapsed[i,:] = np.mean(y[low:high,:], axis=0)
    y = y_collapsed
    labels = dataset.y[range(0,dataset.y.shape[0],af)]
else:
    labels = dataset.y
Explanation: This notebook is going to look at the mistakes made by the chosen model. Specifically, it's going to look at the log loss for each class on the validation set and produce a Hinton diagram for all the classes to show the confusion matrix for this model. Then, it will find the class performing worst and plot example images from this class after processing.
Aiming for it to be written in such a way that the model file chosen can just be swapped in, so we can try it on different models.
Setup and prediction
Doing the exact same thing as in the notebook on Holdout set testing, creating a set of predictions:
End of explanation
import sklearn.metrics
# split predictions by class and calculate log loss
results = {}
uniform = {}
for c in range(121):
    class_mask = labels.ravel() == c
    l = np.zeros(y.shape[0])
    l[np.where(class_mask)] = 1
    p = np.zeros((y.shape[0],2))
    p[:,1] = sum(class_mask)/len(class_mask)
    p[:,0] = 1-p[:,1]
    uniform[c] = sklearn.metrics.log_loss(l,p)
    p[:,1] = y[:,c]
    p[:,0] = 1-p[:,1]
    results[c] = sklearn.metrics.log_loss(l,p)
Explanation: Log loss by class
Creating a bar chart of log loss by class:
End of explanation
plt.plot(range(121),[results[c] for c in range(121)], c="red", label="Network")
plt.plot(range(121),[uniform[c] for c in range(121)], c="blue", label="Prior")
plt.yscale("log")
l=plt.legend()
Explanation: Comparing the score of the network against a uniform benchmark on a scatter plot:
End of explanation
differences = np.array([uniform[c]-results[c] for c in range(121)])
plt.plot(range(121),differences, c="green")
plt.title("Difference between network and uniform benchmark across classes")
plt.yscale("log")
Explanation: May be better to take the improvement on uniform as a score:
End of explanation
worst = np.where(differences == max(differences))[0][0]
print("Worst performance on: {0}".format(settings.classes[worst]))
Explanation: Finding out which is the worst:
End of explanation
inds = []
confdict = {}
for i,w in enumerate(labels == worst):
    if w:
        r = y[i,:]
        predicted = np.where(r == max(r))[0]
        if predicted != worst:
            inds.append(i)
            print("{0} confused with: {1}".format(settings.classes[worst],
                                                  settings.classes[predicted]))
            confdict[i] = str(settings.classes[predicted][0].upper()
                              +settings.classes[predicted][1:])
inds = np.array(inds)
inds
example_images = dataset.get_topological_view(dataset.X[inds*8,:]).reshape(-1,48,48)
confdict
%opts Image style(cmap='gray')
channels = hl.Image(example_images[0,:,:],label=confdict[inds[0]])
for i,j in enumerate(inds[1:]):
    channels = channels + hl.Image(example_images[i],label=confdict[j])
print("These are all trichodesmium_puffs, labels indicate what they were confused with.")
channels
confusion = []
for r in y[inds,:]:
    predicted = np.where(r == max(r))[0][0]
    confusion.append(predicted)
channels = None
for c in list(set(confusion)):
    np.random.seed(42)
    cinds = np.where(labels == c)[0]
    cinds.setflags(write=True)
    np.random.shuffle(cinds)
    cinds = cinds[:4]
    c_examples = dataset.get_topological_view(
                        dataset.X[cinds*8,:]).reshape(4,48,48)
    cs = str(settings.classes[c])
    cs = cs[:1].upper() + cs[1:]
    for i in range(4):
        if not channels:
            channels = hl.Image(c_examples[i,:,:], label=cs)
        else:
            channels = channels + hl.Image(c_examples[i,:,:], label=cs)
channels
Explanation: What's it being confused with?
End of explanation
N = y.shape[0]
logloss = lambda x: -(1./N)*np.log(x[0][x[1]])[0]
Explanation: Worst Datapoints
Now it's probably worth plotting whcih datapoints the network is gettings confused about most, when it's getting confused. So, we find the datapoints with the highest log loss.
End of explanation
sum(map(logloss,zip(y,labels)))
sklearn.metrics.log_loss(labels,y)
Explanation: Checking I've written this right:
End of explanation
ll = []
for i,(p,l) in enumerate(zip(y,labels)):
    ll.append((i,logloss((p,l))))
ll.sort(key=lambda x: x[-1])
# interesting index to happen
worst = ll[-1][0]
ll[-1]
Explanation: Now we can iterate over all the datapoints and find the worst:
End of explanation
real = dataset.get_topological_view(dataset.X[worst*8:(worst*8)+1,:]).squeeze()
realname = settings.classes[labels[worst][0]]
# capitalise for holoviews...
realname = str(realname[0].upper() + realname[1:])
confused = np.where(y[worst] == max(y[worst]))[0][0]
np.where(labels == confused)[0]
settings.classes[confused]
cexamples = dataset.get_topological_view(dataset.X[np.where(labels == confused)[0]*8,:]).squeeze()
channels = hl.Image(real,group=realname)
for c in cexamples:
    cs = str(settings.classes[confused])
    cs = cs[:1].upper() + cs[1:]
    channels += hl.Image(c, group=cs)
channels
Explanation: Now, we want to plot this index along with examples of what it's being confused with. First, find the class it's being confused with and get some indices for that:
End of explanation
rawX,_ = neukrill_net.utils.load_rawdata(dataset.settings.image_fnames,classes=settings.classes)
cexamples = [rawX[i] for i in np.where(labels == confused)[0]]
real = rawX[worst]
channels = hl.Image(real,group=realname)
for c in cexamples:
    cs = str(settings.classes[confused])
    cs = cs[:1].upper() + cs[1:]
    channels += hl.Image(c, group=cs)
channels
Explanation: I guess the question is, how is the network supposed to be able to tell the difference in this case? More resolution?
Plotting the above with raw images:
End of explanation
# have to discretize y... not comfortable with this
discy = []
for p in y:
    discy.append(np.where(p == max(p))[0][0])
discy = np.array(discy)
discy
labels.ravel()
confmat = sklearn.metrics.confusion_matrix(labels.ravel(),discy)
confmat
np.sum(confmat,axis=1)
confmat = confmat.astype(np.float)
for i,r in enumerate(np.sum(confmat, axis=1)):
    confmat[i] = confmat[i]*(1/r)
confmat
import neukrill_net.hinton_diagram
neukrill_net.hinton_diagram.hinton(confmat)
Explanation: Yeah, looks like our resizing is losing some of the fine-grained detail. Although, that protist fuzzy olive that was the worst has changed relatively little; it was already fuzzy.
Confusion Matrix
We'd like to plot a confusion matrix for these predictions. To do this, we're going to want to use a Hinton plot, because otherwise it's going to be difficult to interpret the 121 by 121 matrix by eye. First, Scikit-learn will make the confusion matrix array for us:
End of explanation
# removing diagonal elements:
diags = np.where(np.eye(len(settings.classes)) == 1)
ndconfmat = confmat[:]
ndconfmat[diags[0],diags[1]] = 0
neukrill_net.hinton_diagram.hinton(ndconfmat)
# which elements are the biggest?
for m in np.sort(ndconfmat.ravel())[-10:]:
    print("{0} at {1}".format(m,np.where(ndconfmat==m)))
Explanation: Where are the big false positives? We can make a new array just including those, in a configuration we know?
End of explanation
c10inds = np.where(labels==10)[0]
c11inds = np.where(labels==11)[0]
c10name = str(settings.classes[10][0].upper()+settings.classes[10][1:])
c11name = str(settings.classes[11][0].upper()+settings.classes[11][1:])
example = dataset.get_topological_view(
    dataset.X[c10inds[-2]*8:(c10inds[-2]*8)+1,:]).squeeze()
channels = hl.Image(example, label=c10name)
for i,j in zip(c10inds[:10],c11inds[:10]):
    # because I want them interleaved
    for k in ((i, c10name),(j, c11name)):
        example = dataset.get_topological_view(
            dataset.X[k[0]*8:(k[0]*8)+1,:]).squeeze()
        channels += hl.Image(example, group=k[1])
channels
Explanation: So confusion is greatest between classes 10 and 11? We could plot examples of those two classes:
End of explanation
<END_TASK>