{"action":"created","comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/comments/39339571","id":39339571,"diff_hunk":"@@ -0,0 +1,53 @@\n+\"\"\"\n+=====================================\n+Visualization of MLP weights on MNIST\n+=====================================\n+\n+Sometimes looking at the learned coefficients of a neural network can provide\n+inside into the learning behavior. For example if weights look unstructured,\n+maybe a weight was not used at all, or if very large coefficients exist, maybe\n+regularization was too low or the learning rate too high.\n+\n+This example shows how to plot some of the first layer weights in a\n+MLPClassifier trained on the MNIST dataset.\n+\n+The input data consists of 28x28 pixel handwritten digits, leading to 784\n+features in the dataset. Therefore the first layer weight have the shape (784,\n+hidden_layer_sizes[0]).  We can therefore visualize a single column of the\n+weight matrix as a 28x28 pixel image.\n+\n+To make the example run faster, we use very few hidden units, and train only\n+for a very short time. Training longer would result in much smoother weights.\n+\"\"\"\n+print(__doc__)\n+\n+import matplotlib.pyplot as plt\n+from sklearn.datasets import fetch_mldata\n+from sklearn.neural_network import MLPClassifier\n+\n+mnist = fetch_mldata(\"MNIST original\")\n+# rescale the data, use the traditional train/test split\n+X, y = mnist.data / 255., mnist.target\n+X_train, X_test = X[:60000], X[60000:]\n+y_train, y_test = y[:60000], y[60000:]\n+\n+#mlp = MLPClassifier(hidden_layer_sizes=(100, 100), max_iter=400, alpha=1e-4,\n+#                    algorithm='sgd', verbose=10, tol=1e-4, random_state=1)\n+mlp = MLPClassifier(hidden_layer_sizes=(50,), max_iter=10, alpha=1e-4,","path":"examples/neural_networks/plot_mnist_filters.py","position":36,"original_position":36,"commit_id":"29bf44cb43a8498403adb803c04e4246f03541a1","original_commit_id":"3d2b9fbb975113e980b8e1058aa72e71982c2026","user":{"login":"amueller","id":449558,"avatar_url":"https://avatars.githubusercontent.com/u/449558?v=3","gravatar_id":"","url":"https://api.github.com/users/amueller","html_url":"https://github.com/amueller","followers_url":"https://api.github.com/users/amueller/followers","following_url":"https://api.github.com/users/amueller/following{/other_user}","gists_url":"https://api.github.com/users/amueller/gists{/gist_id}","starred_url":"https://api.github.com/users/amueller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amueller/subscriptions","organizations_url":"https://api.github.com/users/amueller/orgs","repos_url":"https://api.github.com/users/amueller/repos","events_url":"https://api.github.com/users/amueller/events{/privacy}","received_events_url":"https://api.github.com/users/amueller/received_events","type":"User","site_admin":false},"body":"I have no strong opinion but maybe leave it as it was? Not sure.","created_at":"2015-09-12T17:35:23Z","updated_at":"2015-09-12T17:35:23Z","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5214#discussion_r39339571","pull_request_url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5214","_links":{"self":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/comments/39339571"},"html":{"href":"https://github.com/scikit-learn/scikit-learn/pull/5214#discussion_r39339571"},"pull_request":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5214"}}},"pull_request":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5214","id":44245444,"html_url":"https://github.com/scikit-learn/scikit-learn/pull/5214","diff_url":"https://github.com/scikit-learn/scikit-learn/pull/5214.diff","patch_url":"https://github.com/scikit-learn/scikit-learn/pull/5214.patch","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5214","number":5214,"state":"open","locked":false,"title":"Mlp with adam, nesterov's momentum, early stopping","user":{"login":"glennq","id":5950453,"avatar_url":"https://avatars.githubusercontent.com/u/5950453?v=3","gravatar_id":"","url":"https://api.github.com/users/glennq","html_url":"https://github.com/glennq","followers_url":"https://api.github.com/users/glennq/followers","following_url":"https://api.github.com/users/glennq/following{/other_user}","gists_url":"https://api.github.com/users/glennq/gists{/gist_id}","starred_url":"https://api.github.com/users/glennq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glennq/subscriptions","organizations_url":"https://api.github.com/users/glennq/orgs","repos_url":"https://api.github.com/users/glennq/repos","events_url":"https://api.github.com/users/glennq/events{/privacy}","received_events_url":"https://api.github.com/users/glennq/received_events","type":"User","site_admin":false},"body":"Based on #3204 and #3939.\r\n\r\nCode and results for timing and performance benchmarks on MNIST, 20 New Groups and RCV1 can be found at https://gist.github.com/glennq/44ca8b66770430ee10f9\r\n\r\nBasically my observation is that adam performs consistently well. Early stopping cannot help to improve performance for adam, but can reduce training time significantly. In other updating schemes, adaptive learning rate (divide by 5 if not improving) with nesterov's momentum also lines just after adam, followed by constant learning rate with nesterov's momentum. The problem with adaptive learning rate though, is that training time is significantly longer.\r\n\r\nAlso, there are cases when early stopping can increase training time, which I think is due to the fact that the heuristic we use (stop if loss not improving over best loss by tol for more than two consecutive iterations) can be affected if training loss has large variation but validation score is more smoothly improving. \r\nThis affects update schemes using momentum, but it seems to be not a problem for adam.\r\n\r\nThe following is a visualized classifier comparison modified from http://scikit-learn.org/dev/auto_examples/classification/plot_classifier_comparison.html\r\n\r\n![figure_1](https://cloud.githubusercontent.com/assets/5950453/9690764/60b1c5c8-530c-11e5-83f7-1095741a170c.png)\r\n\r\nI'll still work on benchmarking small datasets such as digits and iris, and probably plot training/validation loss for each iterations for more insights.","created_at":"2015-09-04T18:04:07Z","updated_at":"2015-09-12T17:35:23Z","closed_at":null,"merged_at":null,"merge_commit_sha":"d2233778bc7fdef95db564e79678c5bd74cee5a8","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5214/commits","review_comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5214/comments","review_comment_url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/comments{/number}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5214/comments","statuses_url":"https://api.github.com/repos/scikit-learn/scikit-learn/statuses/29bf44cb43a8498403adb803c04e4246f03541a1","head":{"label":"glennq:mlp_refactoring","ref":"mlp_refactoring","sha":"29bf44cb43a8498403adb803c04e4246f03541a1","user":{"login":"glennq","id":5950453,"avatar_url":"https://avatars.githubusercontent.com/u/5950453?v=3","gravatar_id":"","url":"https://api.github.com/users/glennq","html_url":"https://github.com/glennq","followers_url":"https://api.github.com/users/glennq/followers","following_url":"https://api.github.com/users/glennq/following{/other_user}","gists_url":"https://api.github.com/users/glennq/gists{/gist_id}","starred_url":"https://api.github.com/users/glennq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glennq/subscriptions","organizations_url":"https://api.github.com/users/glennq/orgs","repos_url":"https://api.github.com/users/glennq/repos","events_url":"https://api.github.com/users/glennq/events{/privacy}","received_events_url":"https://api.github.com/users/glennq/received_events","type":"User","site_admin":false},"repo":{"id":41526478,"name":"scikit-learn","full_name":"glennq/scikit-learn","owner":{"login":"glennq","id":5950453,"avatar_url":"https://avatars.githubusercontent.com/u/5950453?v=3","gravatar_id":"","url":"https://api.github.com/users/glennq","html_url":"https://github.com/glennq","followers_url":"https://api.github.com/users/glennq/followers","following_url":"https://api.github.com/users/glennq/following{/other_user}","gists_url":"https://api.github.com/users/glennq/gists{/gist_id}","starred_url":"https://api.github.com/users/glennq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/glennq/subscriptions","organizations_url":"https://api.github.com/users/glennq/orgs","repos_url":"https://api.github.com/users/glennq/repos","events_url":"https://api.github.com/users/glennq/events{/privacy}","received_events_url":"https://api.github.com/users/glennq/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/glennq/scikit-learn","description":"scikit-learn main repo","fork":true,"url":"https://api.github.com/repos/glennq/scikit-learn","forks_url":"https://api.github.com/repos/glennq/scikit-learn/forks","keys_url":"https://api.github.com/repos/glennq/scikit-learn/keys{/key_id}","collaborators_url":"https://api.github.com/repos/glennq/scikit-learn/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/glennq/scikit-learn/teams","hooks_url":"https://api.github.com/repos/glennq/scikit-learn/hooks","issue_events_url":"https://api.github.com/repos/glennq/scikit-learn/issues/events{/number}","events_url":"https://api.github.com/repos/glennq/scikit-learn/events","assignees_url":"https://api.github.com/repos/glennq/scikit-learn/assignees{/user}","branches_url":"https://api.github.com/repos/glennq/scikit-learn/branches{/branch}","tags_url":"https://api.github.com/repos/glennq/scikit-learn/tags","blobs_url":"https://api.github.com/repos/glennq/scikit-learn/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/glennq/scikit-learn/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/glennq/scikit-learn/git/refs{/sha}","trees_url":"https://api.github.com/repos/glennq/scikit-learn/git/trees{/sha}","statuses_url":"https://api.github.com/repos/glennq/scikit-learn/statuses/{sha}","languages_url":"https://api.github.com/repos/glennq/scikit-learn/languages","stargazers_url":"https://api.github.com/repos/glennq/scikit-learn/stargazers","contributors_url":"https://api.github.com/repos/glennq/scikit-learn/contributors","subscribers_url":"https://api.github.com/repos/glennq/scikit-learn/subscribers","subscription_url":"https://api.github.com/repos/glennq/scikit-learn/subscription","commits_url":"https://api.github.com/repos/glennq/scikit-learn/commits{/sha}","git_commits_url":"https://api.github.com/repos/glennq/scikit-learn/git/commits{/sha}","comments_url":"https://api.github.com/repos/glennq/scikit-learn/comments{/number}","issue_comment_url":"https://api.github.com/repos/glennq/scikit-learn/issues/comments{/number}","contents_url":"https://api.github.com/repos/glennq/scikit-learn/contents/{+path}","compare_url":"https://api.github.com/repos/glennq/scikit-learn/compare/{base}...{head}","merges_url":"https://api.github.com/repos/glennq/scikit-learn/merges","archive_url":"https://api.github.com/repos/glennq/scikit-learn/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/glennq/scikit-learn/downloads","issues_url":"https://api.github.com/repos/glennq/scikit-learn/issues{/number}","pulls_url":"https://api.github.com/repos/glennq/scikit-learn/pulls{/number}","milestones_url":"https://api.github.com/repos/glennq/scikit-learn/milestones{/number}","notifications_url":"https://api.github.com/repos/glennq/scikit-learn/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/glennq/scikit-learn/labels{/name}","releases_url":"https://api.github.com/repos/glennq/scikit-learn/releases{/id}","created_at":"2015-08-28T04:12:24Z","updated_at":"2015-08-28T04:12:30Z","pushed_at":"2015-09-11T21:34:40Z","git_url":"git://github.com/glennq/scikit-learn.git","ssh_url":"git@github.com:glennq/scikit-learn.git","clone_url":"https://github.com/glennq/scikit-learn.git","svn_url":"https://github.com/glennq/scikit-learn","homepage":"http://scikit-learn.sourceforge.net","size":88087,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"scikit-learn:master","ref":"master","sha":"cda140e25ac2c42ecdb7fbc527e35193b3adf6b3","user":{"login":"scikit-learn","id":365630,"avatar_url":"https://avatars.githubusercontent.com/u/365630?v=3","gravatar_id":"","url":"https://api.github.com/users/scikit-learn","html_url":"https://github.com/scikit-learn","followers_url":"https://api.github.com/users/scikit-learn/followers","following_url":"https://api.github.com/users/scikit-learn/following{/other_user}","gists_url":"https://api.github.com/users/scikit-learn/gists{/gist_id}","starred_url":"https://api.github.com/users/scikit-learn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scikit-learn/subscriptions","organizations_url":"https://api.github.com/users/scikit-learn/orgs","repos_url":"https://api.github.com/users/scikit-learn/repos","events_url":"https://api.github.com/users/scikit-learn/events{/privacy}","received_events_url":"https://api.github.com/users/scikit-learn/received_events","type":"Organization","site_admin":false},"repo":{"id":843222,"name":"scikit-learn","full_name":"scikit-learn/scikit-learn","owner":{"login":"scikit-learn","id":365630,"avatar_url":"https://avatars.githubusercontent.com/u/365630?v=3","gravatar_id":"","url":"https://api.github.com/users/scikit-learn","html_url":"https://github.com/scikit-learn","followers_url":"https://api.github.com/users/scikit-learn/followers","following_url":"https://api.github.com/users/scikit-learn/following{/other_user}","gists_url":"https://api.github.com/users/scikit-learn/gists{/gist_id}","starred_url":"https://api.github.com/users/scikit-learn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scikit-learn/subscriptions","organizations_url":"https://api.github.com/users/scikit-learn/orgs","repos_url":"https://api.github.com/users/scikit-learn/repos","events_url":"https://api.github.com/users/scikit-learn/events{/privacy}","received_events_url":"https://api.github.com/users/scikit-learn/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/scikit-learn/scikit-learn","description":"scikit-learn: machine learning in Python","fork":false,"url":"https://api.github.com/repos/scikit-learn/scikit-learn","forks_url":"https://api.github.com/repos/scikit-learn/scikit-learn/forks","keys_url":"https://api.github.com/repos/scikit-learn/scikit-learn/keys{/key_id}","collaborators_url":"https://api.github.com/repos/scikit-learn/scikit-learn/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/scikit-learn/scikit-learn/teams","hooks_url":"https://api.github.com/repos/scikit-learn/scikit-learn/hooks","issue_events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/events{/number}","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/events","assignees_url":"https://api.github.com/repos/scikit-learn/scikit-learn/assignees{/user}","branches_url":"https://api.github.com/repos/scikit-learn/scikit-learn/branches{/branch}","tags_url":"https://api.github.com/repos/scikit-learn/scikit-learn/tags","blobs_url":"https://api.github.com/repos/scikit-learn/scikit-learn/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/scikit-learn/scikit-learn/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/scikit-learn/scikit-learn/git/refs{/sha}","trees_url":"https://api.github.com/repos/scikit-learn/scikit-learn/git/trees{/sha}","statuses_url":"https://api.github.com/repos/scikit-learn/scikit-learn/statuses/{sha}","languages_url":"https://api.github.com/repos/scikit-learn/scikit-learn/languages","stargazers_url":"https://api.github.com/repos/scikit-learn/scikit-learn/stargazers","contributors_url":"https://api.github.com/repos/scikit-learn/scikit-learn/contributors","subscribers_url":"https://api.github.com/repos/scikit-learn/scikit-learn/subscribers","subscription_url":"https://api.github.com/repos/scikit-learn/scikit-learn/subscription","commits_url":"https://api.github.com/repos/scikit-learn/scikit-learn/commits{/sha}","git_commits_url":"https://api.github.com/repos/scikit-learn/scikit-learn/git/commits{/sha}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/comments{/number}","issue_comment_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments{/number}","contents_url":"https://api.github.com/repos/scikit-learn/scikit-learn/contents/{+path}","compare_url":"https://api.github.com/repos/scikit-learn/scikit-learn/compare/{base}...{head}","merges_url":"https://api.github.com/repos/scikit-learn/scikit-learn/merges","archive_url":"https://api.github.com/repos/scikit-learn/scikit-learn/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/scikit-learn/scikit-learn/downloads","issues_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues{/number}","pulls_url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls{/number}","milestones_url":"https://api.github.com/repos/scikit-learn/scikit-learn/milestones{/number}","notifications_url":"https://api.github.com/repos/scikit-learn/scikit-learn/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/labels{/name}","releases_url":"https://api.github.com/repos/scikit-learn/scikit-learn/releases{/id}","created_at":"2010-08-17T09:43:38Z","updated_at":"2015-09-12T16:06:26Z","pushed_at":"2015-09-12T17:20:39Z","git_url":"git://github.com/scikit-learn/scikit-learn.git","ssh_url":"git@github.com:scikit-learn/scikit-learn.git","clone_url":"https://github.com/scikit-learn/scikit-learn.git","svn_url":"https://github.com/scikit-learn/scikit-learn","homepage":"http://scikit-learn.org","size":134248,"stargazers_count":7457,"watchers_count":7457,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":4418,"mirror_url":null,"open_issues_count":756,"forks":4418,"open_issues":756,"watchers":7457,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5214"},"html":{"href":"https://github.com/scikit-learn/scikit-learn/pull/5214"},"issue":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5214"},"comments":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5214/comments"},"review_comments":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5214/comments"},"review_comment":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5214/commits"},"statuses":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/statuses/29bf44cb43a8498403adb803c04e4246f03541a1"}}}}