Theano: How to give training data to a neural network
<p>I'm trying to create a simple multilayer Perceptron (MLP) for "logical and" in Theano.
There is one layer between input and output. The structure is this:</p>

<p>2 value input -> multiply with weights, add bias -> softmax -> 1 value output</p>

<p>The change in dimension is caused by the weights matrix.</p>

<p>The implementation is based on this tutorial: <a href="http://deeplearning.net/tutorial/logreg.html" rel="nofollow">http://deeplearning.net/tutorial/logreg.html</a></p>

<p>This is my class for the Layer:</p>

<pre><code>class Layer():
"""
this is a layer in the mlp
it's not meant to predict the outcome hence it does not compute a loss.
apply the functions for negative log likelihood = cost on the output of the last layer
"""

def __init__(self, input, n_in, n_out):
    self.W = theano.shared(
            value=numpy.zeros(
                    (n_in, n_out),
                    dtype=theano.config.floatX
            ),
            name="W",
            borrow=True
    )
    self.b = theano.shared(
            value=numpy.zeros((n_in
                               , n_out),
                              dtype=theano.config.floatX),
            name="b",
            borrow=True
    )

    self.output = T.nnet.softmax(T.dot(input, self.W) + self.b)
    self.params = (self.W, self.b)
    self.input = input
</code></pre>

<p>The class is meant to be modular. Instead of just one layer, I want to be able to add multiple layers.
Therefore the functions for prediction, cost and errors are outside of the class (as opposed to the tutorial):</p>

<pre><code>def y_pred(output):
    return T.argmax(output, axis=1)


def negative_log_likelihood(output, y):
    return -T.mean(T.log(output)[T.arange(y.shape[0]), y])


def errors(output, y):
    # check if y has same dimension of y_pred
    if y.ndim != y_pred(output).ndim:
        raise TypeError(
                'y should have the same shape as self.y_pred',
                ('y', y.type, 'y_pred', y_pred(output).type)
        )
    # check if y is of the correct datatype
    if y.dtype.startswith('int'):
        # the T.neq operator returns a vector of 0s and 1s, where 1
        # represents a mistake in prediction
        return T.mean(T.neq(y_pred(output), y))
    else:
        raise NotImplementedError()
</code></pre>

<p>The logical and has 4 trainingcases:</p>

<ul>
<li>[0,0] -> 0</li>
<li>[1,0] -> 0</li>
<li>[0,1] -> 0</li>
<li>[1,1] -> 1</li>
</ul>

<p>Here are the setup of the classifier and the functions for training and evaluating:</p>

<pre><code>data_x = numpy.matrix([[0, 0],
                       [1, 0],
                       [0, 1],
                       [1, 1]])

data_y = numpy.array([0,
                      0,
                      0,
                      1])

train_set_x = theano.shared(numpy.asarray(data_x,
                         dtype=theano.config.floatX),
                         borrow=True)

train_set_y = T.cast(theano.shared(numpy.asarray(data_y,
                         dtype=theano.config.floatX),
                         borrow=True),"int32")

x = T.vector("x",theano.config.floatX)  # data
y = T.ivector("y")  # labels

classifier = Layer(input=x, n_in=2, n_out=1)

cost = negative_log_likelihood(classifier.output, y)

g_W = T.grad(cost=cost, wrt=classifier.W)
g_b = T.grad(cost=cost, wrt=classifier.b)
index = T.lscalar()

learning_rate = 0.15

updates = [
    (classifier.W, classifier.W - learning_rate * g_W),
    (classifier.b, classifier.b - learning_rate * g_b)
]

train_model = theano.function(
        inputs=[index],
        outputs=cost,
        updates=updates,
        givens={
            x: train_set_x[index],
            y: train_set_y[index]
        }
)
validate_model = theano.function(
        inputs=[index],
        outputs=classifier.errors(y),
        givens={
            x: train_set_x[index],
            y: train_set_y[index]
        }
)
</code></pre>

<p>I tried to follow the conventions. Each row in the data matrix is a training sample. Each training sample is matched to the correct output.
Unfortunately the code breaks. I can't interpret the error message. What did I do wrong ?
Error:</p>

<blockquote>
  <p>TypeError: Cannot convert Type TensorType(int32, scalar) (of Variable Subtensor{int64}.0) into Type TensorType(int32, vector). You can try to manually convert Subtensor{int64}.0 into a TensorType(int32, vector).</p>
</blockquote>

<p>This error occurs deep in the Theano code. The conflicting line in my program is:</p>

<pre><code>train_model = theano.function(
        inputs=[index],
        outputs=cost,
        updates=updates,
        givens={
            x: train_set_x[index],
            y: train_set_y[index]      # &lt;---------------HERE
        }
)
</code></pre>

<p>Apparently there is a mismatch between the dimensions of y and the training data.
My complete code on pastebin: <a href="http://pastebin.com/U5jYitk2" rel="nofollow">http://pastebin.com/U5jYitk2</a>
The complete error message on pastebin: <a href="http://pastebin.com/hUQJhfNM" rel="nofollow">http://pastebin.com/hUQJhfNM</a></p>

<p><strong>Concise question:</strong>
What is the correct way to give training data to an mlp in theano ?
Where is my mistake ?</p>

<p>I copied most of the code of the tutorial. Notable changes (probable causes of the error) are:</p>

<ul>
<li>training data for y is not a matrix. I think this is right, because the output of my network is just a scalar value</li>
<li>input of the first layer is a vector. This variable is named x.</li>
<li>Access of the training data does not use slicing. In the tutorial the training data is very complex and I find it hard to read the data access code. I believe that x should be a row of the data-matrix. This is how I implemented it.</li>
</ul>

<p><strong>UPDATE:</strong>
I used the code of Amir. Looks very good, thank you.</p>

<p>But it creates an error, too. The last loop is out-of-bounds:</p>

<blockquote>
  <p>/usr/bin/python3.4 /home/lhk/programming/sk/mlp/mlp/Layer.py Traceback
  (most recent call last):   File
  "/usr/local/lib/python3.4/dist-packages/theano/compile/function_module.py",
  line 595, in <strong>call</strong>
      outputs = self.fn() ValueError: y_i value out of bounds</p>
  
  <p>During handling of the above exception, another exception occurred:</p>
  
  <p>Traceback (most recent call last):   File
  "/home/lhk/programming/sk/mlp/mlp/Layer.py", line 113, in 
      train_model(i)   File "/usr/local/lib/python3.4/dist-packages/theano/compile/function_module.py",
  line 606, in <strong>call</strong>
      storage_map=self.fn.storage_map)   File "/usr/local/lib/python3.4/dist-packages/theano/gof/link.py", line 206,
  in raise_with_op
      raise exc_type(exc_value).with_traceback(exc_trace)   File "/usr/local/lib/python3.4/dist-packages/theano/compile/function_module.py",
  line 595, in <strong>call</strong>
      outputs = self.fn() ValueError: y_i value out of bounds Apply node that caused the error: CrossentropySoftmaxArgmax1HotWithBias(Dot22.0,
  b, Elemwise{Cast{int32}}.0) Inputs types: [TensorType(float64,
  matrix), TensorType(float64, vector), TensorType(int32, vector)]
  Inputs shapes: [(1, 1), (1,), (1,)] Inputs strides: [(8, 8), (8,),
  (4,)] Inputs values: [array([[ 0.]]), array([ 0.]), array([1],
  dtype=int32)]</p>
  
  <p>HINT: Re-running with most Theano optimization disabled could give you
  a back-trace of when this node was created. This can be done with by
  setting the Theano flag 'optimizer=fast_compile'. If that does not
  work, Theano optimizations can be disabled with 'optimizer=None'.
  HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint
  and storage map footprint of this apply node.</p>
</blockquote>

<p>Line 113 is this one:</p>

<pre><code>#train the model
for i in range(train_set_x.shape[0].eval()):
    train_model(i)              # &lt;-----------------HERE
</code></pre>

<p>I believe this is because the indexing of the training data uses <code>index</code> and <code>index+1</code>. Why is that necessary ? One row should be one training sample. And one row is <code>train_set_x[index]</code></p>

<p>Edit: I debugged the code. Without slicing it returns a 1d-array, with slicing it's 2d. 1d should be incompatible to the matrix x.</p>

<p>But while I did this, I found another strange problem:
I added this code to look at the effect of the training:</p>

<pre><code>print("before")
print(classifier.W.get_value())
print(classifier.b.get_value())

for i in range(3):
    train_model(i)

print("after")
print(classifier.W.get_value())
print(classifier.b.get_value())

before
[[ 0.]
 [ 0.]]
[ 0.]
after
[[ 0.]
 [ 0.]]
[ 0.]
</code></pre>

<p>This makes sense, since the first three samples have 0 as correct output.
If I change the order and move the training sample (1,1),1 to the front, the program crashes.</p>

<blockquote>
  <p>before [[ 0.]  [ 0.]] [ 0.] Traceback (most recent call last):   File
  "/usr/local/lib/python3.4/dist-packages/theano/compile/function_module.py",
  line 595, in <strong>call</strong>
      outputs = self.fn() ValueError: y_i value out of bounds</p>
  
  <p>During handling of the above exception, another exception occurred:</p>
  
  <p>Traceback (most recent call last):   File
  "/home/lhk/programming/sk/mlp/mlp/Layer.py", line 121, in 
      train_model(i)   File "/usr/local/lib/python3.4/dist-packages/theano/compile/function_module.py",
  line 606, in <strong>call</strong>
      storage_map=self.fn.storage_map)   File "/usr/local/lib/python3.4/dist-packages/theano/gof/link.py", line 206,
  in raise_with_op
      raise exc_type(exc_value).with_traceback(exc_trace)   File "/usr/local/lib/python3.4/dist-packages/theano/compile/function_module.py",
  line 595, in <strong>call</strong>
      outputs = self.fn() ValueError: y_i value out of bounds Apply node that caused the error: CrossentropySoftmaxArgmax1HotWithBias(Dot22.0,
  b, Elemwise{Cast{int32}}.0) Inputs types: [TensorType(float64,
  matrix), TensorType(float64, vector), TensorType(int32, vector)]
  Inputs shapes: [(1, 1), (1,), (1,)] Inputs strides: [(8, 8), (8,),
  (4,)] Inputs values: [array([[ 0.]]), array([ 0.]), array([1],
  dtype=int32)]</p>
  
  <p>HINT: Re-running with most Theano optimization disabled could give you
  a back-trace of when this node was created. This can be done with by
  setting the Theano flag 'optimizer=fast_compile'. If that does not
  work, Theano optimizations can be disabled with 'optimizer=None'.
  HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint
  and storage map footprint of this apply node.</p>
</blockquote>

<p><strong>UPDATE</strong></p>

<p>I installed Python2.7 with Theano and tried to run the code again. The same error occurs. And I added verbose exception handling. Here is the output:</p>

<pre><code>/usr/bin/python2.7 /home/lhk/programming/sk/mlp/mlp/Layer.py
Traceback (most recent call last):
  File "/home/lhk/programming/sk/mlp/mlp/Layer.py", line 113, in &lt;module&gt;
    train_model(i)
  File "/home/lhk/.local/lib/python2.7/site-packages/theano/compile/function_module.py", line 595, in __call__
    outputs = self.fn()
  File "/home/lhk/.local/lib/python2.7/site-packages/theano/gof/link.py", line 485, in streamline_default_f
    raise_with_op(node, thunk)
  File "/home/lhk/.local/lib/python2.7/site-packages/theano/gof/link.py", line 481, in streamline_default_f
    thunk()
  File "/home/lhk/.local/lib/python2.7/site-packages/theano/gof/op.py", line 768, in rval
    r = p(n, [x[0] for x in i], o)
  File "/home/lhk/.local/lib/python2.7/site-packages/theano/tensor/nnet/nnet.py", line 896, in perform
    nll[i] = -row[y_idx[i]] + m + numpy.log(sum_j)
IndexError: index 1 is out of bounds for axis 0 with size 1
Apply node that caused the error: CrossentropySoftmaxArgmax1HotWithBias(Dot22.0, b, Subtensor{int32:int32:}.0)
Inputs types: [TensorType(float64, matrix), TensorType(float64, vector), TensorType(int32, vector)]
Inputs shapes: [(1, 1), (1,), (1,)]
Inputs strides: [(8, 8), (8,), (4,)]
Inputs values: [array([[ 0.]]), array([ 0.]), array([1], dtype=int32)]

Debugprint of the apply node: 
CrossentropySoftmaxArgmax1HotWithBias.0 [@A] &lt;TensorType(float64, vector)&gt; ''   
 |Dot22 [@B] &lt;TensorType(float64, matrix)&gt; ''   
 | |Subtensor{int32:int32:} [@C] &lt;TensorType(float64, matrix)&gt; ''   
 | | |&lt;TensorType(float64, matrix)&gt; [@D] &lt;TensorType(float64, matrix)&gt;
 | | |ScalarFromTensor [@E] &lt;int32&gt; ''   
 | | | |&lt;TensorType(int32, scalar)&gt; [@F] &lt;TensorType(int32, scalar)&gt;
 | | |ScalarFromTensor [@G] &lt;int32&gt; ''   
 | |   |Elemwise{add,no_inplace} [@H] &lt;TensorType(int32, scalar)&gt; ''   
 | |     |&lt;TensorType(int32, scalar)&gt; [@F] &lt;TensorType(int32, scalar)&gt;
 | |     |TensorConstant{1} [@I] &lt;TensorType(int8, scalar)&gt;
 | |W [@J] &lt;TensorType(float64, matrix)&gt;
 |b [@K] &lt;TensorType(float64, vector)&gt;
 |Subtensor{int32:int32:} [@L] &lt;TensorType(int32, vector)&gt; ''   
   |Elemwise{Cast{int32}} [@M] &lt;TensorType(int32, vector)&gt; ''   
   | |&lt;TensorType(float64, vector)&gt; [@N] &lt;TensorType(float64, vector)&gt;
   |ScalarFromTensor [@E] &lt;int32&gt; ''   
   |ScalarFromTensor [@G] &lt;int32&gt; ''   
CrossentropySoftmaxArgmax1HotWithBias.1 [@A] &lt;TensorType(float64, matrix)&gt; ''   
CrossentropySoftmaxArgmax1HotWithBias.2 [@A] &lt;TensorType(int32, vector)&gt; ''   

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.

Process finished with exit code 1
</code></pre>

<p><strong>UPDATE:</strong></p>

<p>I looked at the training data again. Any sample with 1 as label will produce the above error.</p>

<pre><code>data_y = numpy.array([1,
                      1,
                      1,
                      1])
</code></pre>

<p>The above sample labels will crash for every train_model(i) for i in (0,1,2,3).
Apparently there is an interference between indexing the samples and the sample content.</p>

<p><strong>UPDATE:</strong>
The problem is indeed, like Amir's contact indicated, the dimension of the output layer. I had the misconception that I could train the network to encode the output of the function "logical and" directly in the output neuron. While this is certainly possible, this training approach uses the y value indexing to choose the output node which should have the highest value. After changing the output size to two, the code works. And with enough training, the errors for all cases indeed become zero.</p>