{"action":"opened","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/270","labels_url":"https://api.github.com/repos/fchollet/keras/issues/270/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/270/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/270/events","html_url":"https://github.com/fchollet/keras/issues/270","id":90440608,"number":270,"title":"Inconsistent behavior of batch normalization layer","user":{"login":"kofd","id":5904259,"avatar_url":"https://avatars.githubusercontent.com/u/5904259?v=3","gravatar_id":"","url":"https://api.github.com/users/kofd","html_url":"https://github.com/kofd","followers_url":"https://api.github.com/users/kofd/followers","following_url":"https://api.github.com/users/kofd/following{/other_user}","gists_url":"https://api.github.com/users/kofd/gists{/gist_id}","starred_url":"https://api.github.com/users/kofd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kofd/subscriptions","organizations_url":"https://api.github.com/users/kofd/orgs","repos_url":"https://api.github.com/users/kofd/repos","events_url":"https://api.github.com/users/kofd/events{/privacy}","received_events_url":"https://api.github.com/users/kofd/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-06-23T17:09:42Z","updated_at":"2015-06-23T17:09:42Z","closed_at":null,"body":"As you can see by the code below, the behavior of batch normalization seems to be dependent on the batch of samples. Any given sample should always be classified exactly the same (in mode 0 at least). Note that for the single item batch, the output of model.predict(arr1[:1]) is identical to the output of model.predict(arr1[1:]).\r\n\r\n```\r\n>>> import numpy\r\n>>> \r\n>>> from keras.models import Sequential\r\n>>> from keras.layers.core import Dense, Activation\r\n>>> from keras.layers.normalization import BatchNormalization\r\n>>> from keras.optimizers import SGD\r\n>>> \r\n>>> model = Sequential()\r\n>>> model.add(Dense(200,5))\r\n>>> model.add(BatchNormalization(5))\r\n>>> model.add(Activation('softmax'))\r\n>>> sgd = SGD(lr=0.01, decay=3e-6, momentum=0.9, nesterov=False)\r\n>>> model.compile(loss='categorical_crossentropy', optimizer=sgd)\r\n>>> arr1 = numpy.random.rand(2,200)\r\n>>> y1 = numpy.random.rand(2,5)\r\n>>> model.train(arr1,y1)\r\narray(3.693775110217432)\r\n>>> model.predict(arr1)\r\n2/2 [==============================] - 0s\r\narray([[ 0.21000414,  0.20690492,  0.19180691,  0.19093657,  0.20034746],\r\n       [ 0.1894262 ,  0.19338509,  0.20825628,  0.20882832,  0.2001041 ]])\r\n>>> model.predict(arr1[:1])\r\n1/1 [==============================] - 0s\r\narray([[ 0.19959944,  0.20018073,  0.20001223,  0.1998318 ,  0.2003758 ]])\r\n>>> model.predict(arr1[1:])\r\n1/1 [==============================] - 0s\r\narray([[ 0.19959944,  0.20018073,  0.20001223,  0.1998318 ,  0.2003758 ]])\r\n```\r\n\r\nI've read the source code a thousand times and cannot find a reason for this to happen. I've got a working theory that there is an issue with running mean and running std being mutable within get_output(). Those values are also not inspectable as they are not stored as a tensor value:\r\n```\r\n>>> import theano\r\n>>> theano.printing.debugprint(model.layers[1].running_mean)\r\nElemwise{true_div,no_inplace} [@A] ''   \r\n |Sum{axis=[0], acc_dtype=float64} [@B] ''   \r\n | |Elemwise{add,no_inplace} [@C] ''   \r\n |   |dot [@D] ''   \r\n |   | |<TensorType(float64, matrix)> [@E]\r\n |   | |<TensorType(float64, matrix)> [@F]\r\n |   |DimShuffle{x,0} [@G] ''   \r\n |     |<TensorType(float64, vector)> [@H]\r\n |DimShuffle{x} [@I] ''   \r\n   |Subtensor{int64} [@J] ''   \r\n     |Elemwise{Cast{float64}} [@K] ''   \r\n     | |Shape [@L] ''   \r\n     |   |Elemwise{add,no_inplace} [@C] ''   \r\n     |Constant{0} [@M]\r\n```\r\n\r\nThis issue makes batch normalization fairly useless to me as I cannot guarantee that a sample will always generate the same prediction."}}