<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 PyTorch practice with MNIST data
WNixalo - 2018/2/27
0. Imports
Step1:  1. Data Loading
Step2:  2. Network Definition
Step3:  3. Loss Function & Optimizer Definitions
Step4:  4. Training
Aside
Step5:  
Step6:  NOTE
Step7:  okay, that's actually so cool.
Aside
Step8:  Making sure the images match up with their corresponding labels
Step9:  Aside
Step10:  Having trouble now with 1 channel images in pyplot. 3 channel HxWxC is fine for 1 channel, HxW is fine, but both CxHxW and HxWxC fail if C = 1.
7. Testing
7.1 Test Run on Single MiniBatch
Step11:  Sanity Check
Step12:  And the images object being wrapped as a pytorch Variable & sent into the network is itself a torch.FloatTensor -- to answer the question of how do I get data into a PyTorch neuralnet again?
type(images) --> torch.FloatTensor; coming out of dataiter.next() which itself is iter(testloader) which defined in §1.
Step13:  $93.75\%$ accuracy, though this is a notoriously easy dataset.
7.2 Testing on full test dataset
Step15:  Awesome.
8 Custom Data
Step16:  And here we see an immediate jumpy to ~ 81% accuracy on completely new images, thanks to preprocessing -- to make the images look like those the network trained on.
Step17:  Predictions on images without preprocessing
Step19:  Well that was super cool. The Linear/Fully-Connected/Dense Neural Net on it's own is very limited. It can't recognize spatially-correlated features. Maybe it can, but it's ability  to do is ephemeral and a byproduct at best. I can do better with a simple ConvNet.
9. Simple ConvNet
Step20:  9.1 Train ConvNet & ConvNetMod
Step21:  That loss is about 1/3 what I got with the same training regimen on a Linear network.
More training
Step22:  ConvNetMod
Step23:  Wow, nevermind. Introducing the large-receptive-field conv layer in the beginning really helped with the loss.
9.2 Test accuracy of ConvNets
Step24:  technically less accurate, but I know it has a lot more potential than the other method, and will be more resilient.
Aside
Step25:  9.3 Run ConvNets on custom data;
Step26:  ConvNet after 4 epochs at lr=0.01 and 2 epochs at lr=0.05
Step27:  ConvNetMod after 2 epochs of training
Step28:  Aside
Step29:  Although I wonder if I should have preprocessed my written digits to be color inverted. I see a lot of -1's in the PT MNIST data, and +1's in my own. They're both normalized in range $[-1,1]$, making me think that's the case. Luckily it's a simple NumPy or OpenCV operation away, and can just be done in the initial image-loading phase.
I may do so without updating this Aside.
Z. Misc
Aside
Step30:  Okay, that looks like my problem. The images inside ... ahhhh... okay. Far as I can tell, here's what's up
Step31:  Misc
<ASSISTANT_TASK:>
Python Code:
# %reload_ext autoreload
# %autoreload 2
%matplotlib inline
import torch
import torchvision
import numpy as np
# import mnist_loader
# train, valid, test = mnist_loader.load_data(path='data/mnist/')
Explanation: PyTorch practice with MNIST data
WNixalo - 2018/2/27
0. Imports
End of explanation
# torchvision datasets are PIL.Image images of range [0,1]. Must trsfm them 
# to Tensors of normalized range [-1,1]
transform = torchvision.transforms.Compose(
    [torchvision.transforms.ToTensor(),
     torchvision.transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))]
)
# I have no idea how this works
trainset = torchvision.datasets.MNIST(root='./data/mnist/', train=True,
                                      download=True, transform=transform)
testset  = torchvision.datasets.MNIST(root='./data/mnnist/', train=False, 
                                      download=False, transform=transform)
# ^ already downloaded w/ trainset; doesnt need to be `download=True`
trainloader = torch.utils.data.DataLoader(trainset, batch_size=32, 
                                          shuffle=True, num_workers=4)
testloader  = torch.utils.data.DataLoader(testset, batch_size=32, 
                                          shuffle=False, num_workers=4)
classes = tuple(str(i+1) for i in range(10))
classes
Explanation: 1. Data Loading
End of explanation
class FullNet(torch.nn.Module):
    def __init__(self, layers):
        super().__init__()
        self.layers = torch.nn.ModuleList([
            torch.nn.Linear(layers[i], layers[i+1]) for i in range(len(layers) - 1)
        ])
    def forward(self, x):
        x = x.view(x.size(0), -1)
        for layer in self.layers:
            layer_x = layer(x)
            x = torch.nn.functional.relu(layer_x) # NOTE we dont use relu as last actvn
        return torch.nn.functional.log_softmax(layer_x, dim=-1) # but we DO take the last layer's output
layers = [1*28*28, 40, 10] # MNIST dims: torch.Size([1, 28, 28]
network = FullNet(layers)
network
[param.numel() for param in network.parameters()]
Explanation: 2. Network Definition
End of explanation
# loss function
criterion = torch.nn.CrossEntropyLoss()
# backprop optimizer
optimizer = torch.optim.SGD(network.parameters(), lr=1e-2, momentum=0.9)
network.parameters
Explanation: 3. Loss Function & Optimizer Definitions
End of explanation
trainloader
# I don't know how to get just one minibatch out of trainloader but this works
for i, data in enumerate(trainloader):
    if i == 0:
        inp, lab = data
        inp, lab = (torch.autograd.Variable(x) for x in (inp,lab))
        
outp = network(inp)
loss = criterion(outp, lab)
for elm in (loss.data, loss.data[0]): print(elm)
Explanation: 4. Training
Aside: example training one minibatch
End of explanation
# some help from: 
# http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#train-the-network
for epoch in range(2):
    running_loss = 0.0
    for i, data in enumerate(trainloader, start=0):
        # get inputs & wrap them in Variable
        inputs, labels = data
        inputs, labels = torch.autograd.Variable(inputs), torch.autograd.Variable(labels)
        
        # zero the parameter gradients
        optimizer.zero_grad()
        
        # forward + backward + optimize
        outputs = network(inputs)             # forward computation step
        loss    = criterion(outputs, labels)  # loss calculation
        loss.backward()                       # backward (backprop) comp.step (gradient calculations)
        optimizer.step()                      # SGD step: update weights
        
        # print statistics
        running_loss += loss.data[0]
        if i % 200 == 199: # print every 200 mini-batches
            print(f'{[epoch+1, i+1]} loss: {running_loss/2000}')
            
print(f'Training Loop Complete')
Explanation: 
End of explanation
# help from:
# http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#test-the-network-on-the-test-data
# oh so can you just wrap the DataLoader in `iter()` and get elements that way? ahhh..
dataiter = iter(testloader)
images, labels = dataiter.next()
# images
def imshow(img):
    img = img / 2 + 0.5 # unnormalize
    npimg = img.numpy()
    try:
        plt.imshow(np.transpose(npimg, (1, 2, 0)))
    except NameError:
        import matplotlib.pyplot as plt  # not sure if Py3.6 supports condtl imports..
        plt.imshow(np.transpose(npimg, (1, 2, 0)))
imshow(torchvision.utils.make_grid(images))
for i in range(len(labels)//8):
    print('  ' + ' '.join(f'{labels[i*8 + j]:3d}' for j in range(8)))
# print(f'Ground Truth: '.join(f'{classes[labels[j]]:5s}' for j in range(4)))
Explanation: NOTE: forgot to divide running loss by 200, not 2000.
5. Visualizing Data
End of explanation
# original
[[labels[i+j] for j in range(8)] for i in range(4)]
# corrected
[[labels[i*8 + j] for j in range(8)] for i in range(4)]
Explanation: okay, that's actually so cool.
Aside: figuring out why the labels were printed wrong (corrected now)
End of explanation
img = images[8]
img = img / 2 + 0.5
img = img.numpy()
img = np.transpose(img, (1,2,0))
img.shape
print(img[:,:,0].shape)
plt.imshow(img[:,:,0],cmap='gray');
labels[8]
Explanation: Making sure the images match up with their corresponding labels:
End of explanation
img = torchvision.utils.make_grid(images)
img = img / 2 + 0.5
img = img.numpy()   # (3, 122, 242)
# plt.imshow(np.transpose(img, (1, 2, 0)))  # (122, 242, 3)
# plt.imshow(np.transpose(img, (0, 1, 2)))  # (3, 122, 242) ## invalid dims
# plt.imshow(np.transpose(img, (0, 1, 2))[0])  # (122, 242) ## this works instead
# So, plt needs HxWxC, but will work with CxHxW iff C=1
# Our images from DataLoader are CxHxW
# Can use img = np.transpose(img, (1,2,0)) to convert CxHxW --> HxWxC
img = images[-1]
img = torchvision.utils.make_grid(img)
print(img.shape)
print(np.transpose(img, (1,2,0)).shape)
img = images[-1]
img = img / 2 + 0.5
img = img.numpy()
# img = np.transpose(img, (1,2,0))
# img.shape
plt.imshow(img[0], cmap='gray') # see: https://matplotlib.org/examples/color/colormaps_reference.html
print(img.shape)
Explanation: Aside: understanding some matrix stuff for images
End of explanation
# test run on first minibatch:
dataiter = iter(testloader)
images, labels = dataiter.next()
outputs = network(torch.autograd.Variable(images))
# btw, it seems getting further into iterators is where itertools comes in:
# https://stackoverflow.com/a/3267069
Explanation: Having trouble now with 1 channel images in pyplot. 3 channel HxWxC is fine for 1 channel, HxW is fine, but both CxHxW and HxWxC fail if C = 1.
7. Testing
7.1 Test Run on Single MiniBatch
End of explanation
outputs.shape
Explanation: Sanity Check: output of neuralnet for 1 minibatch of 32 images for 10 classes is a $32 \times 10$ tensor.
End of explanation
_, predicted = torch.max(outputs.data, 1)
# output tensor is 32x10. That's 32 rows of values in 10 columns.
# torch.max(outputs.data, 1) specifies finding maximum value along axis-1 (ie: columnwise)
# torch.max(..) returns: (value, index)
imshow(torchvision.utils.make_grid(images))
# accuracy:
print(sum(labels == predicted)/len(labels)) # sum of torch.ByteTensor size 32
# predictions:
[[predicted[i*8 + j] for j in range(8)] for i in range(4)]
Explanation: And the images object being wrapped as a pytorch Variable & sent into the network is itself a torch.FloatTensor -- to answer the question of how do I get data into a PyTorch neuralnet again?
type(images) --> torch.FloatTensor; coming out of dataiter.next() which itself is iter(testloader) which defined in §1.
End of explanation
correct = 0
total = 0
for datum in testloader:
    images, labels  = datum
    outputs = network(torch.autograd.Variable(images))
    _, predicted = torch.max(outputs.data, 1)
    
    total += labels.size(0)  # labels.size() --> torch.Size([size])
    correct += (predicted == labels).sum()  # method of torch.ByteTensor
print(f'MNIST dataset accuracy: {100*np.round(correct/total, 4)}%\n{total} images.')
Explanation: $93.75\%$ accuracy, though this is a notoriously easy dataset.
7.2 Testing on full test dataset
End of explanation
import cv2
import os
flist = os.listdir('data/digits/')
flist = ['data/digits/' + f for f in flist]
flist.remove('data/digits/digits.png')
flist
custom_labels = [f.split('-')[-1].split('.')[0] for f in flist]
# remove 'a' & 'b'
for i,elm in enumerate(custom_labels):
    if len(elm) > 1:
        custom_labels[i] = custom_labels[i][0]
custom_labels = [int(λ) for λ in custom_labels]
# labels should be of type: [torch.LongTensor of size 16] size acc. to num labels
custom_labels = torch.Tensor(custom_labels)  # I think this'll do
images = [cv2.imread(f) for f in flist]
# reference for what datatype images will be:
# [torch.FloatTensor of size 16x1x28x28] --- last minibatch: 10000 % 32 = 16
# array of images from disk
images = [cv2.imread(f) for f in flist]
# resize images to 28x28
images = [cv2.resize(img, (28,28), interpolation=cv2.INTER_AREA) for img in images]
# convert 3-channel png to Grayscale (opencv loads as bgr)
images = [cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) for img in images]
# invert grayscale (to be in line w/ MNIST data)
images = [255 - img for img in images]
# convert HxW dims to HxWxC for PyTorch usage later
images = [np.expand_dims(img, -1) for img in images]
## NOTE: need remove last dim to view in pyplot.imshow via: images[idx][:,:,0]
## another way to do resizings
# get transformation foreach image (x,y : c,r)
# trsfs = [[28/img.shape[1], 28/img.shape[0]] for img in images]
# img = cv2.resize(img, None, fx=28/img.shape[1], fy=28/img.shape[0], interpolation=cv2.INTER_AREA)
## create PyTorch dataset class
# from: http://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class
class custom_dataset(torch.utils.data.Dataset):
    def __init__(self, data_array, transform=None): # going to want to put inputs&labels in as zipped tuples
        self.data = data_array
        self.transform=transform
    
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, idx):
        Return a dict of image & label
        image = self.data[idx][0]
        label = self.data[idx][1]
        
        if self.transform:
            image = self.transform(image) # wait, does this alter self.data[idx][0] ??
    
#         return {'image':image, 'label':label} ## I don't see key names in PT MNIST loader
        return (image, label)
## create dataloader
transform = torchvision.transforms.Compose(
    [torchvision.transforms.ToTensor(),
     torchvision.transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))]
)
custom_data = custom_dataset(list(zip(images, custom_labels)), transform=transform)
custom_dataloader = torch.utils.data.DataLoader(custom_data, batch_size=32,
                                                shuffle=False, num_workers=4)
## create batch-iterator of dataloader
custom_dataiter = iter(custom_dataloader)
## Test Network on Data
indata, labels  = custom_dataiter.next()
indata  = torch.autograd.Variable(indata)
outputs = network(indata)
_, predictions = torch.max(outputs, 1)
Explanation: Awesome.
8 Custom Data
End of explanation
print('Y*  Y\'\n------')
correct = 0
for elem in zip(labels, predictions.data):
    correct += elem[0]==elem[1]
    print(int(elem[0]), ' ', elem[1])
    
print('------')
total   = len(labels)
print(f'A: {100*np.round((correct/total), 4)}%')
Explanation: And here we see an immediate jumpy to ~ 81% accuracy on completely new images, thanks to preprocessing -- to make the images look like those the network trained on.
End of explanation
print('Y*  Y\'\n------')
correct = 0
for elem in zip(labels, predictions.data):
    correct += elem[0]==elem[1]
    print(int(elem[0]), ' ', elem[1])
    
print('------')
total   = len(labels)
print(f'A: {100*np.round((correct/total), 4)}%')
Explanation: Predictions on images without preprocessing:
Predictions on my own hand-written digits. Zero accuracy, but understandable. This is a standard Linear Neural Net. It's nodes are looking for specific numbers in the input tensor, they're not really capable of spotting spatial features. I expect accuracy to get back to 'normal', ~90% range, after I preprocess to invert black/white.
End of explanation
class ConvNet(torch.nn.Module):
    def __init__(self, layers, c):
        super().__init__()
        self.layers = torch.nn.ModuleList([
            torch.nn.Conv2d(layers[i], layers[i+1], kernel_size=3, stride=2) 
            for i in range(len(layers) - 1)
        ])
#         self.pool = nn.AdaptiveMaxPool2d(1)             # can use this to pool
        self.out = torch.nn.Linear(layers[-1], c)
    
    def forward(self, x):
        for layer in self.layers:
            x = layer(x)
        x = torch.nn.functional.adaptive_avg_pool2d(x, 1) # or this to pool
        x = x.view(x.size(0), -1)
        return torch.nn.functional.log_softmax(self.out(x), dim=-1)
class ConvNetMod(torch.nn.Module):
    'Modern' ConvNet using large-field input conv layer
    def __init__(self, layers, c):
        super().__init__()
        # 'modern' large-field input Conv layer
        self.conv1 = torch.nn.Conv2d(1,10,kernel_size=5, stride=1, padding=2)
        # Conv block
        self.layers = torch.nn.ModuleList([
            torch.nn.Conv2d(layers[i], layers[i+1], kernel_size=3, stride=2) 
            for i in range(len(layers) - 1)
        ])
        # output classification Linear layer
        self.out = torch.nn.Linear(layers[-1], c)
        
    def forward(self, x):
        x = self.conv1(x)
        for layer in self.layers:   # conv layers in convblock
            x = layer(x)
        x = torch.nn.functional.adaptive_max_pool2d(x, 1) # pool after conv block
        x = x.view(x.size(0), -1)                         # reshape final featmap to vector
        return torch.nn.functional.log_softmax(self.out(x), dim=-1) # apply output nonlinearity on final outputs
Explanation: Well that was super cool. The Linear/Fully-Connected/Dense Neural Net on it's own is very limited. It can't recognize spatially-correlated features. Maybe it can, but it's ability  to do is ephemeral and a byproduct at best. I can do better with a simple ConvNet.
9. Simple ConvNet
End of explanation
layers = [1, 20, 40, 80] # this may be too much -- filter size by last layer is ..? 7? 3.5?
c = 10
convnetwork = ConvNet(layers, c)
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(convnetwork.parameters(), lr=0.01, momentum=0.9)
num_epochs = 2
for epoch in range(num_epochs):
    running_loss = 0.0
    
    for i, datum in enumerate(trainloader):
        inputs, labels = datum
        inputs, labels = torch.autograd.Variable(inputs), torch.autograd.Variable(labels)
        
        optimizer.zero_grad()
        
        outputs = convnetwork(inputs)
        loss    = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        
#         print(f'minibatch no. {i+1}  Loss: {loss.data[0]/(i+1)}')
        
        running_loss += loss.data[0]
        if i % 200 == 199: # print every 200 mini-batches
            print(f'{[epoch+1, i+1]} loss: {running_loss/200}')
            running_loss = 0.0
            
print(f'Training Loop Complete')
Explanation: 9.1 Train ConvNet & ConvNetMod:
Is it valid to put put the optimizer and criterion inside the network object? Yeah, right? If I set loss = convnetwork.criterion(outputs, labels), then loss.backward() that'll update the loss within the network object yeah?
$\Longrightarrow$ nah wasn't working the way I had it implemented.
NOTE: lol.. it trains 2x ~ 4x as fast if I correctly set the stride to 2 (had it to 1 at first). Yeah ofc.
ConvNet:
End of explanation
num_epochs = 2
for epoch in range(num_epochs):
    running_loss = 0.0
    
    for i, datum in enumerate(trainloader):
        inputs, labels = datum
        inputs, labels = torch.autograd.Variable(inputs), torch.autograd.Variable(labels)
        
        optimizer.zero_grad()
        
        outputs = convnetwork(inputs)
        loss    = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        
#         print(f'minibatch no. {i+1}  Loss: {loss.data[0]/(i+1)}')
        
        running_loss += loss.data[0]
        if i % 200 == 199: # print every 200 mini-batches
            print(f'{[epoch+1, i+1]} loss: {running_loss/200}')
            running_loss = 0.0
            
print(f'Training Loop Complete')
optimizer = torch.optim.SGD(convnetwork.parameters(), lr=0.005, momentum=0.9)
num_epochs = 2
for epoch in range(num_epochs):
    running_loss = 0.0
    
    for i, datum in enumerate(trainloader):
        inputs, labels = datum
        inputs, labels = torch.autograd.Variable(inputs), torch.autograd.Variable(labels)
        
        optimizer.zero_grad()
        
        outputs = convnetwork(inputs)
        loss    = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        
#         print(f'minibatch no. {i+1}  Loss: {loss.data[0]/(i+1)}')
        
        running_loss += loss.data[0]
        if i % 200 == 199: # print every 200 mini-batches
            print(f'{[epoch+1, i+1]} loss: {running_loss/200}')
            running_loss = 0.0
            
print(f'Training Loop Complete')
Explanation: That loss is about 1/3 what I got with the same training regimen on a Linear network.
More training:
End of explanation
layers = [10, 20, 40, 80] # this may be too much -- filter size by last layer is ..? 7? 3.5?
c = 10
convnetworkmod = ConvNetMod(layers, c)
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(convnetworkmod.parameters(), lr=0.01, momentum=0.9)
num_epochs = 2
for epoch in range(num_epochs):
    running_loss = 0.0
    
    for i, datum in enumerate(trainloader):
        inputs, labels = datum
        inputs, labels = torch.autograd.Variable(inputs), torch.autograd.Variable(labels)
        
        optimizer.zero_grad()
        
        outputs = convnetworkmod(inputs)
        loss    = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        
#         print(f'minibatch no. {i+1}  Loss: {loss.data[0]/(i+1)}')
        
        running_loss += loss.data[0]
        if i % 200 == 199: # print every 200 mini-batches
            print(f'{[epoch+1, i+1]} loss: {running_loss/200}')
            running_loss = 0.0
            
print(f'Training Loop Complete')
Explanation: ConvNetMod:
This wasn't training the first time. I think at 6 layers I'm too deep to effectively train, especially with that large first layer. Maybe it would work better if I used batch norm. Also, I wonder if the benefit of the large initial receptive field is mitigated by the tiny and relatively simple images.
Oh.. it's because I set the optimizer to optimizer = torch.optim.SGD(convnetwork.parameters(), ...) It was optimizing for the wrong network architecture.
End of explanation
correct = 0
total   = 0
for datum in testloader:
    inputs, labels = datum
    inputs         = torch.autograd.Variable(inputs)
    
    outputs = convnetwork(inputs)
    _, predictions = torch.max(outputs.data, 1)
    
    total += labels.size(0)
    correct += (labels == predictions).sum()
    
print(f'MNIST dataset accuracy: {100*np.round(correct/total, 4)}%\n{total} images.')
Explanation: Wow, nevermind. Introducing the large-receptive-field conv layer in the beginning really helped with the loss.
9.2 Test accuracy of ConvNets
End of explanation
labels.size(0)
torch.autograd.Variable(labels).size(0)
len(labels)
len(torch.autograd.Variable(labels))
Explanation: technically less accurate, but I know it has a lot more potential than the other method, and will be more resilient.
Aside: checking how I can access the number of elements in a minibatch via labels:
End of explanation
## NOTE: make sure the custom_dataloader from §8 is in memory
Explanation: 9.3 Run ConvNets on custom data;
End of explanation
## create batch-iterator of dataloader
custom_dataiter = iter(custom_dataloader)
indata, labels  = custom_dataiter.next()
indata  = torch.autograd.Variable(indata)
outputs = convnetwork(indata)
_, predictions = torch.max(outputs, 1)
print('Y*  Y\'\n------')
correct = 0
for elem in zip(labels, predictions.data):
    correct += elem[0]==elem[1]
    print(int(elem[0]), ' ', elem[1])
    
print('------')
total   = len(labels)
print(f'A: {100*np.round((correct/total), 4)}%')
Explanation: ConvNet after 4 epochs at lr=0.01 and 2 epochs at lr=0.05:
End of explanation
custom_dataiter = iter(custom_dataloader)
indata, labels  = custom_dataiter.next()
indata  = torch.autograd.Variable(indata)
outputs = convnetworkmod(indata)
_, predictions = torch.max(outputs, 1)
print('Y*  Y\'\n------')
correct = 0
for elem in zip(labels, predictions.data):
    correct += elem[0]==elem[1]
    print(int(elem[0]), ' ', elem[1])
    
print('------')
total   = len(labels)
print(f'A: {100*np.round((correct/total), 4)}%')
def viewimg(image_array, idx):
    image = image_array[idx]
    image = image[:,:,0]
    image = image/2 + 0.5
    image = 255 - image
    plt.imshow(image, cmap='gray')
idx =0
viewimg(images, idx), print(f'Convnet\'s prediction: {predictions.data[idx]}')
idx =1
viewimg(images, idx), print(f'Convnet\'s prediction: {predictions.data[idx]}')
idx =4
viewimg(images, idx), print(f'Convnet\'s prediction: {predictions.data[idx]}')
idx =6
viewimg(images, idx), print(f'Convnet\'s prediction: {predictions.data[idx]}')
idx =20
viewimg(images, idx), print(f'Convnet\'s prediction: {predictions.data[idx]}')
idx =15
viewimg(images, idx), print(f'Convnet\'s prediction: {predictions.data[idx]}')
idx =19
viewimg(images, idx), print(f'Convnet\'s prediction: {predictions.data[idx]}')
idx =8
viewimg(images, idx), print(f'Convnet\'s prediction: {predictions.data[idx]}')
Explanation: ConvNetMod after 2 epochs of training:
End of explanation
# custom_dataiter.next()
# dataiter = iter(testloader)
# dataiter.next()
Explanation: Aside: Checking that my custom data iterator works exactly the same as the PT MNIST one:
End of explanation
custom_dataiter.next()
img = images[0]
torch.from_numpy(img.transpose((2,0,1)))
np.expand_dims(img, -1).shape
torch.from_numpy(np.expand_dims(img,-1).transpose((2,0,1)))
Explanation: Although I wonder if I should have preprocessed my written digits to be color inverted. I see a lot of -1's in the PT MNIST data, and +1's in my own. They're both normalized in range $[-1,1]$, making me think that's the case. Luckily it's a simple NumPy or OpenCV operation away, and can just be done in the initial image-loading phase.
I may do so without updating this Aside.
Z. Misc
Aside: Getting the DataLoader to work:
End of explanation
test
# custom_data.data[0]
custom_dataiter = iter(custom_data)
custom_dataiter.next()
type(custom_data.data[0][0])
a = [i for i in range(6)]
b = [chr(ord('a')+i) for i in range(6)]
c = list(zip(a,b))
c
ord('a')
torchvision.datasets.mnist??
trainset
# trainset[0]
# torchvision datasets are PIL.Image images of range [0,1]. Must trsfm them 
# to Tensors of normalized range [-1,1]
transform = torchvision.transforms.Compose(
    [torchvision.transforms.ToTensor(),
     torchvision.transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))]
)
# I have no idea how this works
trainset = torchvision.datasets.MNIST(root='./data/mnist/', train=True,
                                      download=True, transform=transform)
testset  = torchvision.datasets.MNIST(root='./data/mnnist/', train=False, 
                                      download=False, transform=transform)
# ^ already downloaded w/ trainset; doesnt need to be `download=True`
trainloader = torch.utils.data.DataLoader(trainset, batch_size=32, 
                                          shuffle=True, num_workers=4)
testloader  = torch.utils.data.DataLoader(testset, batch_size=32, 
                                          shuffle=False, num_workers=4)
images[0].dtype
img = images[0]
img = np.expand_dims(img, -1)
img.shape
# compatibility for torchvision.transforms.Compose() -- expects NumPy ndarrays to be of 
# shape: HxWxC as acc to error:
#      46     if isinstance(pic, np.ndarray):
#      47         # handle numpy array
# ---> 48         img = torch.from_numpy(pic.transpose((2, 0, 1))
images = [np.expand_dims(img, -1) for img in images] # cvt (28,28) --> (28,28,1)
images = [transform(img) for img in images]
custom_loader = torch.utils.data.dataloader.DataLoader((images,custom_labels),
                                    batch_size=32,shuffle=False, num_workers=4)
# testset.test_data
images[0]
img = images[0]
img = img / 2 + 0.5
img = img.numpy()
# img = np.transpose(img, (1,2,0))
plt.imshow(img[0], cmap='gray')
y = network(torch.autograd.Variable(images[0]))
y
torch.max(y, 1)
dataiter = iter(testloader)
images, labels = dataiter.next()
x = dataiter.next()
# x[0][0]
testloader
testset
# custom_labels
custom_labels[0]
custom_dataset = 
Explanation: Okay, that looks like my problem. The images inside ... ahhhh... okay. Far as I can tell, here's what's up:
My images from the dataset are failing to load (ie: the dataloader is failing to retrieve images from the dataset) because they're failing a NumPy transpose operation. PyTorch is looking for MxCxHxW Tensors (M=minibatch), and the assumption is images (I think) will be in HxWxC format $\longrightarrow$ so they get transformed to CxHxW. Issue is my images have only HxW dimensions so this operation fails.
But it looks like if I provide the image in the right format, the transpose operation returns the correct-looking formatted image. Cool.
Why the ahh?  Well inside the MNIST dataset class: torchvision.datasets.mnist, there's a delineation between 'raw' and 'processed' images. The data for MNIST is first downloaded, but they arrive as compressed .gz files. These are then unzipped into a raw folder, then saved to a processed folder. I don't know if PyTorch does its image preprocessing on the images when they are saved to processed or after they're retreived. Anyway, when you pull an item from the dataset, internally it goes and retrieves it from the processed folder, or it initalizes the data array in memory from that folder (unsure which).
End of explanation
for i, layer in enumerate(layers, start=0): # 2nd arg is index start count
    print(i, layer)
len(trainloader)
trainloader.batch_size
trainloader.dataset[0]
trainloader.dataset[0][0]
trainloader.dataset[1][0]
Explanation: Misc:
End of explanation
<END_TASK>