{"action":"opened","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/464","labels_url":"https://api.github.com/repos/fchollet/keras/issues/464/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/464/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/464/events","html_url":"https://github.com/fchollet/keras/issues/464","id":98220850,"number":464,"title":"Incorrect MSE reported on status bar when l2_lambda is not zero.","user":{"login":"sergeyf","id":1874668,"avatar_url":"https://avatars.githubusercontent.com/u/1874668?v=3","gravatar_id":"","url":"https://api.github.com/users/sergeyf","html_url":"https://github.com/sergeyf","followers_url":"https://api.github.com/users/sergeyf/followers","following_url":"https://api.github.com/users/sergeyf/following{/other_user}","gists_url":"https://api.github.com/users/sergeyf/gists{/gist_id}","starred_url":"https://api.github.com/users/sergeyf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sergeyf/subscriptions","organizations_url":"https://api.github.com/users/sergeyf/orgs","repos_url":"https://api.github.com/users/sergeyf/repos","events_url":"https://api.github.com/users/sergeyf/events{/privacy}","received_events_url":"https://api.github.com/users/sergeyf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-07-30T17:24:57Z","updated_at":"2015-07-30T17:24:57Z","closed_at":null,"body":"Consider the following toy example dataset and network:\r\n\r\n```\r\nfrom __future__ import print_function\r\nimport numpy as np\r\nfrom keras.models import Graph\r\nfrom keras.layers.core import Dense\r\nfrom keras.regularizers import l2\r\n\r\n# generate random data\r\nd = 6000\r\nX1 = np.random.random((10000,d))**2\r\nX2 = np.log(np.random.random((10000,d)))\r\nY = (np.dot(X1,np.random.random((d,1))) - np.dot(X2,np.random.random((d,1))))**2\r\nY /= Y.max() # scale to be between 0 and 1\r\n\r\ndata = {'X1':X1, 'X2':X2, 'output':Y}\r\n\r\n# network parameters\r\nd1 = 512\r\nd2 = 256\r\nl2_lambda = 1e-3\r\n\r\n# graph model\r\nmodel = Graph()\r\n\r\n# inputs\r\nmodel.add_input(name='X1', ndim=2)\r\nmodel.add_input(name='X2', ndim=2)\r\n\r\n# X1 dense layer\r\nmodel.add_node(Dense(d, d1, activation='relu',W_regularizer=l2(l2_lambda)), \r\n               name='dense_X1', input='X1')\r\n\r\n# X2 dense layer\r\nmodel.add_node(Dense(d, d1, activation='relu',W_regularizer=l2(l2_lambda)), \r\n               name='dense_X2', input='X2')\r\n\r\n# merging dense layer\r\nmodel.add_node(Dense(2*d1 , d2, activation='relu',W_regularizer=l2(l2_lambda)), \r\n               name='dense_merge', merge_mode=\"concat\", \r\n               inputs=['dense_X1','dense_X2'])\r\n\r\n# output dense layer\r\nmodel.add_node(Dense(d2 , 1, activation='sigmoid',W_regularizer=l2(l2_lambda)), \r\n               name='dense_final',input='dense_merge')      \r\n               \r\nmodel.add_output(name='output', input=\"dense_final\")\r\n\r\nmodel.compile('rmsprop', {\"output\": 'mse'})\r\n```\r\n\r\nFirst, I check the MSE of the network BEFORE it is trained. \r\n\r\n```\r\npredictions = model.predict(data)\r\nprint('MSE before any training:')\r\nprint(np.mean((predictions['output']-Y)**2))\r\n\r\nMSE before any training:\r\n0.444015524597\r\n```\r\n\r\nThe MSE before training is 0.44. So, once we actually start training, we would expect the progress bar to report something in that vicinity, and certainly a number no larger than 1.0 since `Y` was scaled to be between 0 and 1,\r\n\r\nHowever, during the actual training the ```output``` is nonsensical:\r\n\r\n```\r\nhistory = model.fit(data=data, nb_epoch=3, validation_split=0.25)\r\nTrain on 7500 samples, validate on 2500 samples\r\nEpoch 0\r\n7500/7500 [==============================] - 2s - output: 2.1041 - val_output: 0.0096\r\nEpoch 1\r\n7500/7500 [==============================] - 2s - output: 1.6632 - val_output: 0.0096\r\n```\r\n\r\nNote the `output: 2.1041` which is MUCH bigger than the val_output. How can the MSE be this large?\r\n\r\nI have noticed that this bug is not active if we change `l2_lambda` to be 0:\r\n\r\n```\r\nMSE before any training:\r\n0.0487085829439\r\n\r\nTrain on 7500 samples, validate on 2500 samples\r\nEpoch 0\r\n7500/7500 [==============================] - 2s - output: 0.0093 - val_output: 0.0083\r\nEpoch 1\r\n7500/7500 [==============================] - 2s - output: 0.0084 - val_output: 0.0083\r\n```\r\n\r\nAny idea what's going on here?"}}