<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Convolutional networks
This is an annotated version of PyTorch's example MNIST CNN code for how to implement convolutional networks.
The CIFAR-10 classification task is a classic machine learning benchmark. The data includes 50,000 images belonging to 10 classes, and the task is to identify them. Along with MNIST, CIFAR-10 classification is a sort of like a "hello world" for computer vision and convolutional networks, so a solution can be implemented quickly with an off-the-shelf machine learning library.
Since convolutional neural networks have thus far proven to be the best at computer vision tasks, we'll use the PyTorch library to implement a convolutional networks as our solution.
Note, if you have been running these notebooks on a regular laptop without GPU until now, it's going to become more and more difficult to do so. The neural networks we will be training, starting with convolutional networks, will become increasingly memory and processing-intensive and may slow down laptops without good graphics processing. As such, it is recommended that you follow future tutorials inside Google Collab or a sufficiently powerful computer. 
Collab does not automatically include a GPU in it's runtime. Rather by navigating to Edit>Notebook Settings>Hardware Accelerator>GPU you can enable a gpu.
Step1:  Recall that a basic neural network in PyTorch can be set up like this
Step2:  We load CIFAR-10 dataset and reshape them as individual vectors. This is easy to do using TorchVision as all the classes are essentially reshapen already. Here we use the DataLoader to create an instance of the training dataset and the test set. The DataLoader also has an option to shuffle the dataset, which helps avoid overfitting.
Step3:  Let's see some of our samples.
Step4:  We can compile the model using categorical-cross-entropy loss, and train it for 30 epochs.
Step5:  At this point, there is practically no need to evaluate the model as our neural net is simply not cutting it. With a loss function in the millions, we cannot use a single layer simple neural network on images.
Introducing, the convolutional network... The general architecture of a convolutional neural network is this
Step6:  Let's compile the model and test it again.
Step7:  Let's evaluate the model again.
Step8:  58% accuracy is a big improvement on practically nothing! All of that is accomplished in just 2 epochs using convolutional layers and ReLUs. 
Let's try to make the network bigger.
One problem you might notice is that the accuracy of the model is much better on the training set than on the test set. You can see that by monitoring the progress at the end of each epoch above or by evaluating it directly.
This is a symptom of "overfitting".  Our model has probably tried to bend itself a little too well towards predicting the training set but does not generalize very well to unseen data. This is a very common problem. 
It's normal for the training accuracy to be better than the testng accuracy to some degree, because it's hard to avoid for the network to be better at predicting the data it sees. But a 9% difference is too much.
One way of helping this is by doing some regularization. We can add dropout to our model after a few layers.
All this is arguably most essential, albeit timeconsuming, part of designing a neural network
Step9:  61% accuracy! Our best yet. Looks heavily overfit though...
Still a long way to go to beat the record (96%).  We can make a lot of progress by making the network (much) bigger, training for (much) longer and using a lot of little tricks (like data augmentation) but that is beyond the scope of this lesson for now.
Let's also recall how to predict a single value and look at its probabilities.
Step10:  Let's also review here how to save and load trained PyTorch models. It's easy!
<ASSISTANT_TASK:>
Python Code:
import random
import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision
import torchvision.transforms as transforms
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
Explanation: Convolutional networks
This is an annotated version of PyTorch's example MNIST CNN code for how to implement convolutional networks.
The CIFAR-10 classification task is a classic machine learning benchmark. The data includes 50,000 images belonging to 10 classes, and the task is to identify them. Along with MNIST, CIFAR-10 classification is a sort of like a "hello world" for computer vision and convolutional networks, so a solution can be implemented quickly with an off-the-shelf machine learning library.
Since convolutional neural networks have thus far proven to be the best at computer vision tasks, we'll use the PyTorch library to implement a convolutional networks as our solution.
Note, if you have been running these notebooks on a regular laptop without GPU until now, it's going to become more and more difficult to do so. The neural networks we will be training, starting with convolutional networks, will become increasingly memory and processing-intensive and may slow down laptops without good graphics processing. As such, it is recommended that you follow future tutorials inside Google Collab or a sufficiently powerful computer. 
Collab does not automatically include a GPU in it's runtime. Rather by navigating to Edit>Notebook Settings>Hardware Accelerator>GPU you can enable a gpu.
End of explanation
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(32, 64) 
        self.fc2 = nn.Linear(64, 1)
    def forward(self, x):
         x = F.sigmoid(self.fc1(x))
         x = self.fc2(x)
         return x
net = Net().to(device)
Explanation: Recall that a basic neural network in PyTorch can be set up like this:
Note:
The images in CIFAR-10 are 32 by 32. An easy way to troubleshoot the correct input dimension is to encounter a dimension mismatch error. For example, [2x35] x [1024x4] doesn't work, but [2x1024] x [1924x4] does. This is usually as simple as matching the second matrix's first number with the first matrix's second number.
End of explanation
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=128,
                                          shuffle=True, num_workers=8)
testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=128,
                                         shuffle=False, num_workers=8)
classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
Explanation: We load CIFAR-10 dataset and reshape them as individual vectors. This is easy to do using TorchVision as all the classes are essentially reshapen already. Here we use the DataLoader to create an instance of the training dataset and the test set. The DataLoader also has an option to shuffle the dataset, which helps avoid overfitting.
End of explanation
def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()
# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()
# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))
Explanation: Let's see some of our samples.
End of explanation
from torch import optim
optimizer = optim.SGD(net.parameters(), lr=0.01)
criterion = nn.MSELoss()
def fullPass(loader):
    running_loss = 0.0
    for i, data in enumerate(loader, 0):
        inputs, labels = data
        inputs, labels = inputs.to(device), labels.to(device)
        outputs = net(inputs)
        loss = criterion(outputs, labels.float())
        loss.backward()
        optimizer.step()
        # print statistics
        running_loss += loss
        if i % inputs.size()[0] == inputs.size()[0]-1:
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / inputs.size()[0]))
            running_loss = 0.0
net.train()
for epoch in range(30):
    fullPass(trainloader);
print('Finished Training')
Explanation: We can compile the model using categorical-cross-entropy loss, and train it for 30 epochs.
End of explanation
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)
    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
net = Net().to(device)
print(net)
Explanation: At this point, there is practically no need to evaluate the model as our neural net is simply not cutting it. With a loss function in the millions, we cannot use a single layer simple neural network on images.
Introducing, the convolutional network... The general architecture of a convolutional neural network is this:
convolution layers, followed by pooling layers
fully-connected layers
a final fully-connected softmax layer
We'll follow this same basic structure and interweave some other components, such as [dropout](https://en.wikipedia.org/wiki/Dropout_(neural_networks), to improve performance.
To begin, we start with our convolution layers. We first need to specify some architectural hyperparemeters:
How many filters do we want for our convolution layers? Like most hyperparameters, this is chosen through a mix of intuition and tuning. A rough rule of thumb is: the more complex the task, the more filters. (Note that we don't need to have the same number of filters for each convolution layer, but we are doing so here for convenience.)
What size should our convolution filters be? We don't want filters to be too large or the resulting matrix might not be very meaningful. For instance, a useless filter size in this task would be a 28x28 filter since it covers the whole image. We also don't want filters to be too small for a similar reason, e.g. a 1x1 filter just returns each pixel.
What size should our pooling window be? Again, we don't want pooling windows to be too large or we'll be throwing away information. However, for larger images, a larger pooling window might be appropriate (same goes for convolution filters).
We start by designing a neural network with two alternating convolutional and max-pooling layers, followed by a 100-neuron fully-connected layer and a 10-neuron output. We'll have 64 and 32 filters in the two convolutional layers, and make the input shape a full-sized image (32x32x3) instead of an unrolled vector (3072x1).  We also now use ReLU activation units instead of sigmoids, to avoid vanishing gradients.
End of explanation
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
for epoch in range(30):  # loop over the dataset multiple times
    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        inputs, labels = inputs.to(device), labels.to(device)
        # zero the parameter gradients
        optimizer.zero_grad()
        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        # print statistics
        running_loss += loss.item()
        if i % 128 == 127:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 128))
            running_loss = 0.0
print('Finished Training')
Explanation: Let's compile the model and test it again.
End of explanation
correct = 0
total = 0
with torch.no_grad():
    for data in testloader:
        images, labels = data
        images, labels = images.to(device), labels.to(device)
        outputs = net(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))
Explanation: Let's evaluate the model again.
End of explanation
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=0.001)
for epoch in range(5):  # loop over the dataset multiple times
    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data
        inputs, labels = inputs.to(device), labels.to(device)
        # zero the parameter gradients
        optimizer.zero_grad()
        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        # print statistics
        running_loss += loss.item()
        if i % 128 == 127:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 128))
            running_loss = 0.0
print('Finished Training')
correct = 0
total = 0
with torch.no_grad():
    for data in testloader:
        images, labels = data
        images, labels = images.to(device), labels.to(device)
        outputs = net(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()
print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))
Explanation: 58% accuracy is a big improvement on practically nothing! All of that is accomplished in just 2 epochs using convolutional layers and ReLUs. 
Let's try to make the network bigger.
One problem you might notice is that the accuracy of the model is much better on the training set than on the test set. You can see that by monitoring the progress at the end of each epoch above or by evaluating it directly.
This is a symptom of "overfitting".  Our model has probably tried to bend itself a little too well towards predicting the training set but does not generalize very well to unseen data. This is a very common problem. 
It's normal for the training accuracy to be better than the testng accuracy to some degree, because it's hard to avoid for the network to be better at predicting the data it sees. But a 9% difference is too much.
One way of helping this is by doing some regularization. We can add dropout to our model after a few layers.
All this is arguably most essential, albeit timeconsuming, part of designing a neural network: Editting hyperparamenters.
We compile and train again.
Training accuracy will likely lower but our test accuracy will increase.  This is more like what we expect.
Another way of improving performance is to experiment with different optimizers beyond just standard sgd. Let's try to instantiate the same network but use ADAM instead of sgd.
End of explanation
dataiter = iter(testloader)
images, labels = dataiter.next()
# print images
imshow(torchvision.utils.make_grid(images))
print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))
_, predicted = torch.max(outputs, 1)
print('Predicted: ', ' '.join('%5s' % classes[predicted[j]]
                              for j in range(4)))
Explanation: 61% accuracy! Our best yet. Looks heavily overfit though...
Still a long way to go to beat the record (96%).  We can make a lot of progress by making the network (much) bigger, training for (much) longer and using a lot of little tricks (like data augmentation) but that is beyond the scope of this lesson for now.
Let's also recall how to predict a single value and look at its probabilities.
End of explanation
PATH = './cifar_net.pth'
torch.save(net.state_dict(), PATH)
torch.load(PATH)
Explanation: Let's also review here how to save and load trained PyTorch models. It's easy!
End of explanation
<END_TASK>