{"action":"created","issue":{"url":"https://api.github.com/repos/dnouri/nolearn/issues/49","labels_url":"https://api.github.com/repos/dnouri/nolearn/issues/49/labels{/name}","comments_url":"https://api.github.com/repos/dnouri/nolearn/issues/49/comments","events_url":"https://api.github.com/repos/dnouri/nolearn/issues/49/events","html_url":"https://github.com/dnouri/nolearn/issues/49","id":58899755,"number":49,"title":"Cannot match validation loss from training when calculating after training","user":{"login":"run2","id":3134655,"avatar_url":"https://avatars.githubusercontent.com/u/3134655?v=3","gravatar_id":"","url":"https://api.github.com/users/run2","html_url":"https://github.com/run2","followers_url":"https://api.github.com/users/run2/followers","following_url":"https://api.github.com/users/run2/following{/other_user}","gists_url":"https://api.github.com/users/run2/gists{/gist_id}","starred_url":"https://api.github.com/users/run2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/run2/subscriptions","organizations_url":"https://api.github.com/users/run2/orgs","repos_url":"https://api.github.com/users/run2/repos","events_url":"https://api.github.com/users/run2/events{/privacy}","received_events_url":"https://api.github.com/users/run2/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-02-25T13:16:09Z","updated_at":"2015-02-25T14:31:13Z","closed_at":null,"body":"Hi\r\n\r\nI am facing this strange problem - which I am struggling for ages now.\r\n\r\nI am training a net - using images and labels. So my regression is false and uselabelencoder is true and in train, test split, the StratifiedKFold gets invoked etc.\r\n\r\nNow, I am using EarlyStopping. When my net trains and exists out due to early stopping, the best weights are loaded into the net (as in the example for face rectangles). I can see the validation loss on that best epoch\r\n\r\nAfter training is finished, I use the net and get the validation set again - using the same StratifiedKFold logic (I made sure that the indices and labels are exactly the same of the validation set used inside the training loop). I now use predict_proba (I have tried with _output_layer.get_output too). and a normal numpy method to get the loss (negative log likelihood) - and the validation loss is different from the best validation loss during training. And that difference is very big (I understand there might be some decimals off here and there for stuff being calculated on the GPU). It seems - the loss I am getting is more close to the last loss I see during the training. Not the best loss.\r\n\r\nNow - just to add some points\r\n1) I have made sure there is no transformation or augmentation of images in real time - both while training and predicting\r\n2) I have made sure that the code I am using to calculate the loss is exactly the same as the negative_log_likelihood function. \r\n3) I have made sure that the weights are being copied over only when (and finally once at the least) the valid loss is decreasing.\r\n\r\nCan some one tell me what I am missing here ? or is there some problem lurking somewhere ?\r\n\r\nThanks\r\nRegards\r\n"},"comment":{"url":"https://api.github.com/repos/dnouri/nolearn/issues/comments/75968830","html_url":"https://github.com/dnouri/nolearn/issues/49#issuecomment-75968830","issue_url":"https://api.github.com/repos/dnouri/nolearn/issues/49","id":75968830,"user":{"login":"dnouri","id":56370,"avatar_url":"https://avatars.githubusercontent.com/u/56370?v=3","gravatar_id":"","url":"https://api.github.com/users/dnouri","html_url":"https://github.com/dnouri","followers_url":"https://api.github.com/users/dnouri/followers","following_url":"https://api.github.com/users/dnouri/following{/other_user}","gists_url":"https://api.github.com/users/dnouri/gists{/gist_id}","starred_url":"https://api.github.com/users/dnouri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dnouri/subscriptions","organizations_url":"https://api.github.com/users/dnouri/orgs","repos_url":"https://api.github.com/users/dnouri/repos","events_url":"https://api.github.com/users/dnouri/events{/privacy}","received_events_url":"https://api.github.com/users/dnouri/received_events","type":"User","site_admin":false},"created_at":"2015-02-25T14:31:13Z","updated_at":"2015-02-25T14:31:13Z","body":"If you're not sure that you're calculating the loss right, maybe you should try and call your numpy version and the Theano version used by the net with the same values, and verify that they produce the same output.\r\n\r\nHere's an implementation that I have lying around:\r\n\r\n```python\r\nimport scipy as sp\r\n\r\ndef logloss(y_true, y_pred):\r\n    epsilon = 1e-18\r\n    y_pred = sp.maximum(epsilon, y_pred)\r\n    y_pred = sp.minimum(1 - epsilon, y_pred)\r\n    ll = (sum(y_true * sp.log(y_pred) +\r\n              sp.subtract(1, y_true) *\r\n              sp.log(sp.subtract(1, y_pred)))\r\n          )\r\n    ll = ll * -1.0 / len(y_true)\r\n    return ll\r\n```"}}