Does Stochastic Gradient Descent even work with TensorFlow?
<p>I designed a MLP, fully connected, with 2 hidden and one output layer.
I get a nice learning curve if I use batch or mini-batch gradient descent.</p>

<p>But a straight line while performing Stochastic Gradient Descent (violet)
<a href="https://i.stack.imgur.com/YCSEu.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/YCSEu.png" alt="enter image description here"></a></p>

<p>What did I get wrong?</p>

<p>In my understanding, I do stochastic gradient descent with Tensorflow, if I provide just one train/learn example each train step, like:</p>

<pre><code>X = tf.placeholder("float", [None, amountInput],name="Input")
Y = tf.placeholder("float", [None, amountOutput],name="TeachingInput")
...
m, i = sess.run([merged, train_op], feed_dict={X:[input],Y:[label]})
</code></pre>

<p>Whereby input is a 10-component vector and label is a 20-component vector.</p>

<p>For testings I run 1000 iterations, each iterations contains one of 50 prepared train/learn example.
I expected an overfittet nn. But as you see, it doesn't learn :(</p>

<p>Because the nn will perform in an online-learning environment, a mini-batch oder batch gradient descent isn't an option.</p>

<p>thanks for any hints.</p>