<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Lasagne Tutorial
This tutorial assumes basic knowledge of Theano.  Here is a Theano tutorial if you need to get up to speed.
Theano is increadibly useful for compiling and automatically differentiating symbolic expressions transparently on a CPU or GPU.  While it is designed with the application of large neural networks in mind, it has relatively little functionality towards that end.  Lasagne is a Python module built on top of Theano which provides useful blocks which make building neural network models simple.  It has been designed to extend Theano's functionality, so it generally follows Theano's conventions and methods typically accept and return Theano expressions.  In this way, it makes constructing commonly used network structures easy but also allows for arbitrary/unconventional models.  It's also meant to provide a reference implementation which is highly optimized.
Development of Lasagne is ongoing and it is still in pre-release stages.  However, it's been built up enough that there is plenty of useful code and new features and are constantly being added.  It's currently being developed by a diverse group of researchers with different applications in mind, which ensures that it is both generic and coherent.
Step1:  Toy example
As a toy example to demonstrate functionality, we'll train a standard multi-layer perceptron on a simple synthetic two-dimensional four-class dataset.
Step2:  Ingredients
Step3:  DenseLayer
The DenseLayer is the basic building block of the neural network
Step4:  get_output
To actually compute the Theano expression of a stack of Layer subclass instances, use the get_output function of the final layer.  As mentioned above, with no arguments, get_output will compute the output given the input variable input_var of the InputLayer at the base of the network.  You can also pass a Theano symbolic variable to get the output of the network with respect to that variable instead.  N.b. - get_output will soon be changed to a global function instead of a layer method, see Lasagne/#182
Step5:  Tasting
Step6:  Baking
Step7:  Real-world example (MNIST ConvNet)
The above example illustrates the simplest possible usage of lasagne.  Fortunately, it's also useful for real-world problems.  Here, we'll train a convolutional network to classify MNIST digits.  This example is based on the mnist_conv.py script included with Lasagne.
Step8:  ConvNet Input
In lasagne, the convention for a 2D convolutional network is that the data's shape throughout the network is (n_examples, n_channels, width, height).  Since MNIST digits has a single channel (they're grayscale images), n_channels = 1 for the input layer; if we were dealing with RGB images we'd have n_channels = 3.  Within the network, n_channels is the number of filter kernels of each layer.
Conveniently, we can make the first dimension (the "number of example" dimension) variable.  This comes in handy when you pass your network training examples in minibatches, but want to evaluate the output on the entire validation or test set.  This is designated by setting the first entry of the shape passed to the InputLayer to None.
Step9:  Convolutional layers
The basic 2D convolutional layer in Lasagne is Conv2DLayer.  This uses Theano's built-in convolution operation to convolve a collection 2D filter against the last two dimensions of its inputs.  Lasagne also provides hooks to the cuda-convnet and cuDNN convolution backends; see lasagne.layers.cuda_convnet and lasagne.layers.dnn respectively.
Note on parameter initialization
The initialization used for each parameter in each layer can be a Numpy ndarray, a Theano shared variable, or an Initializer subclass from lasagne.init.  Many common initialization schemes are included in Lasagne for convenience.  For example, below we'll be initializing the convolutional layer weights using the approach proposed by He et. al. in "Delving Deep into Rectifiers
Step10:  Pooling layers
2D max pooling is straightforward
Step11:  Dense layers
We'll be using a single hidden layer and a dense output layer with dropout in between.  As is the convention in convnets, the hidden layer will use a ReLU nonlinearity.
Step12:  Dropout
Dropout in Lasagne is implemented as a Layer subclass.  By placing a DropoutLayer between layers, the connections between the two layers will randomly be dropped.  As we'll see later, setting get_output or get_loss's keyword argument deterministic to True will make the DropoutLayer act as a simple pass-through, which is useful when computing the output of the network after training.
Step13:  Objectives, updates, and training
This part of the code will look very similar to above - we'll define an objective, compute some updates, use the updates to compile Theano functions, then use the functions to train the network and compute its output given some input.  Since this problem is a little harder than our toy example, we'll use ADADELTA, a fancier stochastic optimization technique, described in ADADELTA
<ASSISTANT_TASK:>
Python Code:
# Lasagne is pre-release, so it's interface is changing.
# Whenever there's a backwards-incompatible change, a warning is raised.
# Let's ignore these for the course of the tutorial
import warnings
warnings.filterwarnings('ignore', module='lasagne')
import theano
import theano.tensor as T
import lasagne
import numpy as np
import sklearn.datasets
from __future__ import print_function
import os
import matplotlib.pyplot as plt
%matplotlib inline
import IPython.display
IPython.display.Image("http://static-vegetariantimes.s3.amazonaws.com/wp-content/uploads/2009/03/10851medium.jpg")
Explanation: Lasagne Tutorial
This tutorial assumes basic knowledge of Theano.  Here is a Theano tutorial if you need to get up to speed.
Theano is increadibly useful for compiling and automatically differentiating symbolic expressions transparently on a CPU or GPU.  While it is designed with the application of large neural networks in mind, it has relatively little functionality towards that end.  Lasagne is a Python module built on top of Theano which provides useful blocks which make building neural network models simple.  It has been designed to extend Theano's functionality, so it generally follows Theano's conventions and methods typically accept and return Theano expressions.  In this way, it makes constructing commonly used network structures easy but also allows for arbitrary/unconventional models.  It's also meant to provide a reference implementation which is highly optimized.
Development of Lasagne is ongoing and it is still in pre-release stages.  However, it's been built up enough that there is plenty of useful code and new features and are constantly being added.  It's currently being developed by a diverse group of researchers with different applications in mind, which ensures that it is both generic and coherent.
End of explanation
# Generate synthetic data
N_CLASSES = 4
X, y = sklearn.datasets.make_classification(n_features=2, n_redundant=0,
                                            n_classes=N_CLASSES, n_clusters_per_class=1)
# Convert to theano floatX
X = X.astype(theano.config.floatX)
# Labels should be ints
y = y.astype('int32')
# Make a scatter plot where color encodes class
plt.scatter(X[:, 0], X[:, 1], c=y)
Explanation: Toy example
As a toy example to demonstrate functionality, we'll train a standard multi-layer perceptron on a simple synthetic two-dimensional four-class dataset.
End of explanation
# First, construct an input layer.
# The shape parameter defines the expected input shape, which is just the shape of our data matrix X.
l_in = lasagne.layers.InputLayer(shape=X.shape)
Explanation: Ingredients: layers
What is lasagne made out of?  A stack of layers of noodles, sauce, cheeses, etc.  Just like making lasagne, constructing a network in lasagne typically involves stacking Layer subclass instances.  Much of Lasagne's functionality centers around subclasses of the Layer class.  A typical Layer subclass is an implementation of some kind of commonly used neural network layer, and contains the layer's parameters as well as methods for computing the Theano expression of the network given a Theano tensor variable used as input.  Most Layers are initialized with a pointer to their input layer.  The output of the network can be generated using the final layer's get_output method, which recursively computes a symbolic Theano expression for the output of all layers given an input.  
InputLayer
The class InputLayer is a special layer type which stops the recurision and allows the user to input actual data into the network.  It also, for convenience, instantiates a Theano tensor variable input_var of the right type given the shape you provide it.  The input_var variable is used by default to compute the symbolic Theano expression of the network (see below).
End of explanation
# We'll create a network with two dense layers: A tanh hidden layer and a softmax output layer.
l_hidden = lasagne.layers.DenseLayer(
    # The first argument is the input layer
    l_in,
    # This defines the layer's output dimensionality
    num_units=10,
    # Various nonlinearities are available
    nonlinearity=lasagne.nonlinearities.tanh)
# For our output layer, we'll use a dense layer with a softmax nonlinearity.
l_output = lasagne.layers.DenseLayer(
    l_hidden, num_units=N_CLASSES, nonlinearity=lasagne.nonlinearities.softmax)
Explanation: DenseLayer
The DenseLayer is the basic building block of the neural network: It computes a linear mix of the input $x$ using a weight matrix $W$ and a bias vector $b$, and then applies a nonlinearity $\sigma$, yielding $\sigma(Wx + b)$.  The DenseLayer class keeps track of the parameters and how to use them to compute this expression.
End of explanation
net_output = l_output.get_output()
Explanation: get_output
To actually compute the Theano expression of a stack of Layer subclass instances, use the get_output function of the final layer.  As mentioned above, with no arguments, get_output will compute the output given the input variable input_var of the InputLayer at the base of the network.  You can also pass a Theano symbolic variable to get the output of the network with respect to that variable instead.  N.b. - get_output will soon be changed to a global function instead of a layer method, see Lasagne/#182
End of explanation
# As a loss function, we'll use Theano's categorical_crossentropy function.
# This allows for the network output to be class probabilities,
# but the target output to be class labels.
true_output = T.ivector('true_output')
objective = lasagne.objectives.Objective(
    l_output,
    # categorical_crossentropy computes the cross-entropy loss where the network
    # output is class probabilities and the target value is an integer denoting the class.
    loss_function=lasagne.objectives.categorical_crossentropy)
# get_loss computes a Theano expression for the objective, given a target variable
# By default, it will use the network's InputLayer input_var, which is what we want.
loss = objective.get_loss(target=true_output)
Explanation: Tasting: objectives
When making a lasagna, you need a way to decide when it's ready to eat.  A talented lasagne cook can do this by tasting it.  Machine learning practitioners use objective functions to decide when their neural network is ready to use, i.e. has been successfuly trained.  lasagne's objectives.Objective class provides a convenient way to compute a symbolic Theano expression for an objective function.  Usually this involves comparing the output of a network to a true value.
End of explanation
# Retrieving all parameters of the network is done using get_all_params,
# which recursively collects the parameters of all layers connected to the provided layer.
all_params = lasagne.layers.get_all_params(l_output)
# Now, we'll generate updates using Lasagne's SGD function
updates = lasagne.updates.sgd(loss, all_params, learning_rate=1)
# Finally, we can compile Theano functions for training and computing the output.
# Note that because loss depends on the input variable of our input layer,
# we need to retrieve it and tell Theano to use it.
train = theano.function([l_in.input_var, true_output], loss, updates=updates)
get_output = theano.function([l_in.input_var], net_output)
# Train (bake?) for 100 epochs
for n in xrange(100):
    train(X, y)
# Compute the predicted label of the training data.
# The argmax converts the class probability output to class label
y_predicted = np.argmax(get_output(X), axis=1)
# Plot incorrectly classified points as black dots
plt.scatter(X[:, 0], X[:, 1], c=(y != y_predicted), cmap=plt.cm.gray_r)
# Compute and display the accuracy
plt.title("Accuracy: {}%".format(100*np.mean(y == y_predicted)))
Explanation: Baking: updates
After assembling a lasagna, you need to bake it until it's ready to eat.  Getting a neural network ready to use is done by optimizing its parameters with respect to the objective.  Lasagne crucially provides functionality for constructing updates to optimize the network's parameters according to some objective (i.e. train it).  This makes it easy to, for example, train a network with stochastic gradient descent (or something more fancy like AdaGrad).  Computing the updates typically involves collecting the network's parameters using get_all_params then using a function in lasagne.updates.  The resulting updates list can then be fed into a theano.function to train the network.
End of explanation
# We'll use the load_data function from the mnist.py example
from mnist import _load_data
data = _load_data()
# Convert the data from the pickle file into a dict
# The keys will be 'train', 'valid', and 'test'
subsets = ['train', 'valid', 'test']
# Each entry will be a dict with 'X' and 'y' entries
# for the data and labels respectively.
dataset = {}
for (subset_data, subset_labels), subset_name in zip(data, subsets):
    # The data is provided in the shape (n_examples, 784)
    # where 784 = width*height = 28*28
    # We need to reshape for convolutional layer shape conventions - explained below!
    subset_data = subset_data.reshape(
        (subset_data.shape[0], 1, 28, 28))
    dataset[subset_name] = {
        # We need to use data matrices of dtype theano.config.floatX
        'X': subset_data.astype(theano.config.floatX),
        # Labels are integers
        'y': subset_labels.astype(np.int32)}
# Plot an example digit with its label
plt.imshow(dataset['train']['X'][0][0], interpolation='nearest', cmap=plt.cm.gray)
plt.title("Label: {}".format(dataset['train']['y'][0]))
plt.gca().set_axis_off()
Explanation: Real-world example (MNIST ConvNet)
The above example illustrates the simplest possible usage of lasagne.  Fortunately, it's also useful for real-world problems.  Here, we'll train a convolutional network to classify MNIST digits.  This example is based on the mnist_conv.py script included with Lasagne.
End of explanation
# We'll determine the input shape from the first example from the training set.
input_shape = dataset['train']['X'][0].shape
l_in = lasagne.layers.InputLayer(
    shape=(None, input_shape[0], input_shape[1], input_shape[2]))
Explanation: ConvNet Input
In lasagne, the convention for a 2D convolutional network is that the data's shape throughout the network is (n_examples, n_channels, width, height).  Since MNIST digits has a single channel (they're grayscale images), n_channels = 1 for the input layer; if we were dealing with RGB images we'd have n_channels = 3.  Within the network, n_channels is the number of filter kernels of each layer.
Conveniently, we can make the first dimension (the "number of example" dimension) variable.  This comes in handy when you pass your network training examples in minibatches, but want to evaluate the output on the entire validation or test set.  This is designated by setting the first entry of the shape passed to the InputLayer to None.
End of explanation
# Create the first convolutional layer
l_conv1 = lasagne.layers.Conv2DLayer(
    l_in,
    # Here, we set the number of filters and their size.
    num_filters=32, filter_size=(5, 5),
    # lasagne.nonlinearities.rectify is the common ReLU nonlinearity
    nonlinearity=lasagne.nonlinearities.rectify,
    # Use He et. al.'s initialization
    W=lasagne.init.HeNormal(gain='relu'))
# Other arguments: Convolution type (full, same, or valid) and stride
Explanation: Convolutional layers
The basic 2D convolutional layer in Lasagne is Conv2DLayer.  This uses Theano's built-in convolution operation to convolve a collection 2D filter against the last two dimensions of its inputs.  Lasagne also provides hooks to the cuda-convnet and cuDNN convolution backends; see lasagne.layers.cuda_convnet and lasagne.layers.dnn respectively.
Note on parameter initialization
The initialization used for each parameter in each layer can be a Numpy ndarray, a Theano shared variable, or an Initializer subclass from lasagne.init.  Many common initialization schemes are included in Lasagne for convenience.  For example, below we'll be initializing the convolutional layer weights using the approach proposed by He et. al. in "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification".  This approach initializes the weights by sampling a Gaussian distribution with zero mean and $\sigma = \sqrt{\frac{2}{n_in}}$, where $n_in$ is the number of inputs to the layer.
End of explanation
# Here, we do 2x2 max pooling.  The max pooling layer also supports striding
l_pool1 = lasagne.layers.MaxPool2DLayer(l_conv1, pool_size=(2, 2))
# The second convolution/pooling pair is the same as above.
l_conv2 = lasagne.layers.Conv2DLayer(
    l_pool1, num_filters=32, filter_size=(5, 5),
    nonlinearity=lasagne.nonlinearities.rectify,
    W=lasagne.init.HeNormal(gain='relu'))
l_pool2 = lasagne.layers.MaxPool2DLayer(l_conv2, pool_size=(2, 2))
Explanation: Pooling layers
2D max pooling is straightforward: Use the MaxPool2DLayer.  As with the convolutional layer, there are cuda-convnet and cuDNN versions, too.
End of explanation
l_hidden1 = lasagne.layers.DenseLayer(
    l_pool2, num_units=256, 
    nonlinearity=lasagne.nonlinearities.rectify,
    W=lasagne.init.HeNormal(gain='relu'))
Explanation: Dense layers
We'll be using a single hidden layer and a dense output layer with dropout in between.  As is the convention in convnets, the hidden layer will use a ReLU nonlinearity.
End of explanation
# p is the dropout probability
l_hidden1_dropout = lasagne.layers.DropoutLayer(l_hidden1, p=0.5)
l_output = lasagne.layers.DenseLayer(
    l_hidden1_dropout,
    # The number of units in the softmas output layer is the number of classes.
    num_units=10,
    nonlinearity=lasagne.nonlinearities.softmax)
Explanation: Dropout
Dropout in Lasagne is implemented as a Layer subclass.  By placing a DropoutLayer between layers, the connections between the two layers will randomly be dropped.  As we'll see later, setting get_output or get_loss's keyword argument deterministic to True will make the DropoutLayer act as a simple pass-through, which is useful when computing the output of the network after training.
End of explanation
true_output = T.ivector('true_output')
objective = lasagne.objectives.Objective(l_output,
        loss_function=lasagne.objectives.categorical_crossentropy)
# As mentioned above, when using dropout we should define different losses:
# One for training, one for evaluation.  The training loss should apply dropout,
# while the evaluation loss shouldn't.  This is controlled by setting the deterministic kwarg.
loss_train = objective.get_loss(target=true_output, deterministic=False)
loss_eval = objective.get_loss(target=true_output, deterministic=True)
all_params = lasagne.layers.get_all_params(l_output)
# Use ADADELTA for updates
updates = lasagne.updates.adadelta(loss_train, all_params)
train = theano.function([l_in.input_var, true_output], loss_train, updates=updates)
# This is the function we'll use to compute the network's output given an input
# (e.g., for computing accuracy).  Again, we don't want to apply dropout here
# so we set the deterministic kwarg to True.
get_output = theano.function([l_in.input_var], l_output.get_output(deterministic=True))
# Now, let's train it!  We'll chop the training data into mini-batches,
# and compute the validation accuracy every epoch.
BATCH_SIZE = 100
N_EPOCHS = 10
# Keep track of which batch we're training with
batch_idx = 0
# Keep track of which epoch we're on
epoch = 0
while epoch < N_EPOCHS:
    # Extract the training data/label batch and update the parameters with it
    train(dataset['train']['X'][batch_idx:batch_idx + BATCH_SIZE],
          dataset['train']['y'][batch_idx:batch_idx + BATCH_SIZE])
    batch_idx += BATCH_SIZE
    # Once we've trained on the entire training set...
    if batch_idx >= dataset['train']['X'].shape[0]:
        # Reset the batch index
        batch_idx = 0
        # Update the number of epochs trained
        epoch += 1
        # Compute the network's on the validation data
        val_output = get_output(dataset['valid']['X'])
        # The predicted class is just the index of the largest probability in the output
        val_predictions = np.argmax(val_output, axis=1)
        # The accuracy is the average number of correct predictions
        accuracy = np.mean(val_predictions == dataset['valid']['y'])
        print("Epoch {} validation accuracy: {}".format(epoch, accuracy))
# Hmm... maybe MNIST is a toy example too.
Explanation: Objectives, updates, and training
This part of the code will look very similar to above - we'll define an objective, compute some updates, use the updates to compile Theano functions, then use the functions to train the network and compute its output given some input.  Since this problem is a little harder than our toy example, we'll use ADADELTA, a fancier stochastic optimization technique, described in ADADELTA: An Adaptive Learning Rate Method.  There's plenty of others to choose from in lasagne.updates.
End of explanation
<END_TASK>