<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Problem 1
Random images
Step1:  Problem 2
Mean images
Step2:  Problem 3
Randomize data
Step3:  Problem 4
Number per class
Step4:  OK, so there are about 50000 in each class in the training set
Step5:  And about 1870 in each class in the test set
Problem 5
How much overlap is there between training, validation and test samples?
Step6:  What about near duplicates between datasets? (images that are almost identical)
Step7:  Problem 6
Train a logistic regressor on the image data using 50, 100, 1000 and 5000 training samples.
<ASSISTANT_TASK:>
Python Code:
label_map = list('abcdefghij')
fig,axes = pl.subplots(3,3,figsize=(5,5),sharex=True,sharey=True)
with h5py.File(cache_file, 'r') as f:
    for i in range(9):
        ax = axes.flat[i]
        
        idx = np.random.randint(f['test']['images'].shape[0])
        ax.imshow(f['test']['images'][idx],
                  cmap='Greys', interpolation='nearest')
        ax.set_title(label_map[int(f['test']['labels'][idx])])
Explanation: Problem 1
Random images
End of explanation
# Solution: 
with h5py.File(cache_file, 'r') as f:
    # get a unique list of the classes
    classes = np.unique(f['test']['labels'])
    classes.sort()
    nclasses = len(classes)
    
    images = f['test']['images'][:]
    for i,cls in enumerate(classes):
        fig,ax = pl.subplots(1,1,figsize=(2,2))
        mean_img = images[f['test']['labels'][:] == cls].mean(axis=0) # select all images for a given class, take mean
        ax.imshow(mean_img, cmap='Greys', interpolation='nearest') # greyscale colormap, no interpolation
        ax.set_title(label_map[i])
Explanation: Problem 2
Mean images
End of explanation
def randomize(data, labels):
    permutation = np.random.permutation(labels.shape[0])
    shuffled_data = data[permutation]
    shuffled_labels = labels[permutation]
    return shuffled_data, shuffled_labels
with h5py.File(cache_file, 'r') as f:
    train_dataset, train_labels = randomize(f['train']['images'][:], f['train']['labels'][:])
    test_dataset, test_labels = randomize(f['test']['images'][:], f['test']['labels'][:])
Explanation: Problem 3
Randomize data
End of explanation
np.histogram(train_labels, bins=np.arange(0,nclasses+1,1))
Explanation: Problem 4
Number per class
End of explanation
np.histogram(test_labels, bins=np.arange(0,nclasses+1,1))
Explanation: OK, so there are about 50000 in each class in the training set
End of explanation
n_overlaps = []
# the data has been randomize, so let's just check the first 100 images and assume that
#    is a representative sample
for test_img in test_dataset[:100]: 
    diff = (train_dataset - test_img[None]).sum(axis=-1).sum(axis=-1)
    n_overlap = (diff == 0).sum()
    n_overlaps.append(n_overlap)
print("Typical overlap:", np.median(n_overlaps))
pl.hist(n_overlaps)
Explanation: And about 1870 in each class in the test set
Problem 5
How much overlap is there between training, validation and test samples?
End of explanation
n_overlaps = []
threshold = 1E-2 # define an arbitrary threshold -- play with this
# the data has been randomize, so let's just check the first 100 images and assume that
#    is a representative sample
for test_img in test_dataset[:100]: 
    diff = (train_dataset - test_img[None]).sum(axis=-1).sum(axis=-1)
    n_overlap = (np.abs(diff) < threshold).sum()
    n_overlaps.append(n_overlap)
Explanation: What about near duplicates between datasets? (images that are almost identical)
End of explanation
model = LogisticRegression()
image_size = train_dataset.shape[-1]
subset = 50 # replace with 100, 1000, 5000
idx = np.random.choice(np.arange(train_dataset.shape[0]), size=subset)
train_subset_data = train_dataset[idx].reshape(subset, image_size*image_size)
train_subset_labels = train_labels[idx]
model.fit(train_subset_data, train_subset_labels)
predict_labels = model.predict(test_dataset.reshape(test_dataset.shape[0], image_size*image_size))
(predict_labels != test_labels).sum() / float(test_labels.size)
Explanation: Problem 6
Train a logistic regressor on the image data using 50, 100, 1000 and 5000 training samples.
End of explanation
<END_TASK>