{"action":"created","issue":{"url":"https://api.github.com/repos/vlfeat/matconvnet/issues/306","labels_url":"https://api.github.com/repos/vlfeat/matconvnet/issues/306/labels{/name}","comments_url":"https://api.github.com/repos/vlfeat/matconvnet/issues/306/comments","events_url":"https://api.github.com/repos/vlfeat/matconvnet/issues/306/events","html_url":"https://github.com/vlfeat/matconvnet/issues/306","id":114683574,"number":306,"title":"Training AlexNet from scratch ","user":{"login":"SergeyMilyaev","id":2150186,"avatar_url":"https://avatars.githubusercontent.com/u/2150186?v=3","gravatar_id":"","url":"https://api.github.com/users/SergeyMilyaev","html_url":"https://github.com/SergeyMilyaev","followers_url":"https://api.github.com/users/SergeyMilyaev/followers","following_url":"https://api.github.com/users/SergeyMilyaev/following{/other_user}","gists_url":"https://api.github.com/users/SergeyMilyaev/gists{/gist_id}","starred_url":"https://api.github.com/users/SergeyMilyaev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SergeyMilyaev/subscriptions","organizations_url":"https://api.github.com/users/SergeyMilyaev/orgs","repos_url":"https://api.github.com/users/SergeyMilyaev/repos","events_url":"https://api.github.com/users/SergeyMilyaev/events{/privacy}","received_events_url":"https://api.github.com/users/SergeyMilyaev/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2015-11-02T21:05:10Z","updated_at":"2015-11-17T16:08:05Z","closed_at":null,"body":"Hi,\r\n\r\nDid someone tried to train AlexNet from scratch using MatConvNet? With using cnn_imagenet script from the examples and enabled batch normalization my results are far below the performance of the imagenet-caffe-ref (the plot is attached).\r\n\r\nRegards,\r\nSergey\r\n\r\n![2015-11-03 00-03-53](https://cloud.githubusercontent.com/assets/2150186/10894230/7111796e-81be-11e5-906c-788a3ba17861.png)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/vlfeat/matconvnet/issues/comments/157415534","html_url":"https://github.com/vlfeat/matconvnet/issues/306#issuecomment-157415534","issue_url":"https://api.github.com/repos/vlfeat/matconvnet/issues/306","id":157415534,"user":{"login":"vedaldi","id":81309,"avatar_url":"https://avatars.githubusercontent.com/u/81309?v=3","gravatar_id":"","url":"https://api.github.com/users/vedaldi","html_url":"https://github.com/vedaldi","followers_url":"https://api.github.com/users/vedaldi/followers","following_url":"https://api.github.com/users/vedaldi/following{/other_user}","gists_url":"https://api.github.com/users/vedaldi/gists{/gist_id}","starred_url":"https://api.github.com/users/vedaldi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vedaldi/subscriptions","organizations_url":"https://api.github.com/users/vedaldi/orgs","repos_url":"https://api.github.com/users/vedaldi/repos","events_url":"https://api.github.com/users/vedaldi/events{/privacy}","received_events_url":"https://api.github.com/users/vedaldi/received_events","type":"User","site_admin":false},"created_at":"2015-11-17T16:08:05Z","updated_at":"2015-11-17T16:08:05Z","body":"Hi, yes, overfitting is reduced, but so is the final accuracy it seems. AlexNet + Batch Normalisation should stabilise around 45% error on the validation data (before bypassing batch normalisation and using only centre crops, which should further lower this down to 41-42%).\n\nI have now been training SVMs and CNNs for years and my conclusion is that that even huge overfitting is not necessarily to be avoided at all costs. In fact, all it matters is that the validation/test error goes down and the best results are often obtained when the training data is overfitted.\n\n> On 17 Nov 2015, at 08:32, Sergey Milyaev <notifications@github.com> wrote:\n> \n> Update from my experiments: I started training AlexNet without batch normalization. I observe much less overfitting compare to my previous experiments (the plot is attached). I see that the difference between the train and validation errors is similar to the results from the plots, which were presented in this discussion by @AruniRC <https://github.com/AruniRC>. I suppose that finally a similar performance will be obtained.\n>  <https://cloud.githubusercontent.com/assets/2150186/11206248/c6b05e42-8d1e-11e5-8954-2fba60cb6385.png>\n> —\n> Reply to this email directly or view it on GitHub <https://github.com/vlfeat/matconvnet/issues/306#issuecomment-157308179>.\n> \n\n"}}