{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5214","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5214/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5214/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5214/events","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5214","id":104939312,"number":5214,"title":"Mlp with adam, nesterov's momentum, early stopping","user":{"login":"glennq","id":5950453,"avatar_url":"https://avatars.githubusercontent.com/u/5950453?v=3","gravatar_id":"","url":"https://api.github.com/users/glennq","html_url":"https://github.com/glennq","followers_url":"https://api.github.com/users/glennq/followers","following_url":"https://api.github.com/users/glennq/following{/other_user}","gists_url":"https://api.github.com/users/glennq/gists{/gist_id}","starred_url":"https://api.github.com/users/glennq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glennq/subscriptions","organizations_url":"https://api.github.com/users/glennq/orgs","repos_url":"https://api.github.com/users/glennq/repos","events_url":"https://api.github.com/users/glennq/events{/privacy}","received_events_url":"https://api.github.com/users/glennq/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":28,"created_at":"2015-09-04T18:04:07Z","updated_at":"2015-09-13T03:23:58Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5214","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5214","diff_url":"https://github.com/scikit-learn/scikit-learn/pull/5214.diff","patch_url":"https://github.com/scikit-learn/scikit-learn/pull/5214.patch"},"body":"Based on #3204 and #3939.\r\n\r\nCode and results for timing and performance benchmarks on MNIST, 20 New Groups and RCV1 can be found at https://gist.github.com/glennq/44ca8b66770430ee10f9\r\n\r\nBasically my observation is that adam performs consistently well. Early stopping cannot help to improve performance for adam, but can reduce training time significantly. In other updating schemes, adaptive learning rate (divide by 5 if not improving) with nesterov's momentum also lines just after adam, followed by constant learning rate with nesterov's momentum. The problem with adaptive learning rate though, is that training time is significantly longer.\r\n\r\nAlso, there are cases when early stopping can increase training time, which I think is due to the fact that the heuristic we use (stop if loss not improving over best loss by tol for more than two consecutive iterations) can be affected if training loss has large variation but validation score is more smoothly improving. \r\nThis affects update schemes using momentum, but it seems to be not a problem for adam.\r\n\r\nThe following is a visualized classifier comparison modified from http://scikit-learn.org/dev/auto_examples/classification/plot_classifier_comparison.html\r\n\r\n![figure_1](https://cloud.githubusercontent.com/assets/5950453/9690764/60b1c5c8-530c-11e5-83f7-1095741a170c.png)\r\n\r\nI'll still work on benchmarking small datasets such as digits and iris, and probably plot training/validation loss for each iterations for more insights."},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/139842310","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5214#issuecomment-139842310","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5214","id":139842310,"user":{"login":"glennq","id":5950453,"avatar_url":"https://avatars.githubusercontent.com/u/5950453?v=3","gravatar_id":"","url":"https://api.github.com/users/glennq","html_url":"https://github.com/glennq","followers_url":"https://api.github.com/users/glennq/followers","following_url":"https://api.github.com/users/glennq/following{/other_user}","gists_url":"https://api.github.com/users/glennq/gists{/gist_id}","starred_url":"https://api.github.com/users/glennq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glennq/subscriptions","organizations_url":"https://api.github.com/users/glennq/orgs","repos_url":"https://api.github.com/users/glennq/repos","events_url":"https://api.github.com/users/glennq/events{/privacy}","received_events_url":"https://api.github.com/users/glennq/received_events","type":"User","site_admin":false},"created_at":"2015-09-13T03:23:58Z","updated_at":"2015-09-13T03:23:58Z","body":"How about this:\r\n\r\nThe default algorithm 'adam' works pretty well on relatively large datasets (with thousands of training samples or more) in terms of both training time and validation score.\r\nFor small datasets, however, 'l-bfgs' can converge faster perform better."}}