{"action":"created","comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/comments/43575167","id":43575167,"diff_hunk":"@@ -0,0 +1,69 @@\n+\"\"\"\n+==================================================================\n+Plotting Performance of different hyperparameter selection methods\n+==================================================================\n+\n+In this plot you can see the validation scores of a Ridge regression model\n+along steps of different hyperparameter selection methods.\n+\"\"\"\n+print(__doc__)\n+\n+import matplotlib.pyplot as plt\n+\n+import numpy as np\n+from scipy import stats\n+\n+from sklearn.model_selection import SequentialSearchCV\n+from sklearn.model_selection import GridSearchCV\n+from sklearn.model_selection import RandomizedSearchCV\n+from sklearn.linear_model import Ridge\n+from sklearn.datasets import make_regression\n+\n+# Loading the Digits dataset\n+X, y = make_regression(\n+    n_samples=50, n_features=500, noise=1., random_state=0)\n+\n+# build a classifier\n+clf = Ridge()\n+\n+# specify parameters and distributions to sample from\n+expon = stats.expon(scale=20)\n+expon.random_state = 0  # for reproductible results\n+params = {'alpha': expon}\n+\n+# run randomized search\n+gp_search = SequentialSearchCV(\n+  clf, params, verbose=1, n_iter=20)\n+gp_search.fit(X, y)\n+\n+# retrieve the maximum score at each iteration\n+gp_cum_scores = [np.max(\n+      [gp_search.grid_scores_[j].mean_validation_score for j in range(i)])\n+      for i in range(1, len(gp_search.grid_scores_))]\n+\n+rdm_search = RandomizedSearchCV(\n+  clf, params, random_state=0, n_iter=20)\n+rdm_search.fit(X, y)\n+\n+rdm_cum_scores = [np.max(\n+      [rdm_search.grid_scores_[j].mean_validation_score for j in range(i)])\n+      for i in range(1, len(rdm_search.grid_scores_))]\n+\n+\n+params = {'alpha': np.logspace(-3, 3, 20)}","path":"examples/model_selection/plot_gp_search.py","position":53,"original_position":53,"commit_id":"ae685b88f21426d8b8d63fea97c1a6a30635781e","original_commit_id":"ae685b88f21426d8b8d63fea97c1a6a30635781e","user":{"login":"jmetzen","id":1116263,"avatar_url":"https://avatars.githubusercontent.com/u/1116263?v=3","gravatar_id":"","url":"https://api.github.com/users/jmetzen","html_url":"https://github.com/jmetzen","followers_url":"https://api.github.com/users/jmetzen/followers","following_url":"https://api.github.com/users/jmetzen/following{/other_user}","gists_url":"https://api.github.com/users/jmetzen/gists{/gist_id}","starred_url":"https://api.github.com/users/jmetzen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmetzen/subscriptions","organizations_url":"https://api.github.com/users/jmetzen/orgs","repos_url":"https://api.github.com/users/jmetzen/repos","events_url":"https://api.github.com/users/jmetzen/events{/privacy}","received_events_url":"https://api.github.com/users/jmetzen/received_events","type":"User","site_admin":false},"body":"That's a bit unfair since the good values of alpha are all close to the upper boundary of the value range. If you would use `params = {'alpha': np.logspace(-3, 3, 20)[::-1]}? , grid search would perform very well. Moreover, if we fix the budget to 20 iterations, it doesn't really matter if a good parameter value is found in the first iteration or the last. A more realistic benchmark in my opinion would be to vary n_iter (for all *SearchCV) and compare the scores of the best found value for alpha. ","created_at":"2015-10-31T19:14:29Z","updated_at":"2015-10-31T19:14:29Z","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5491#discussion_r43575167","pull_request_url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5491","_links":{"self":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/comments/43575167"},"html":{"href":"https://github.com/scikit-learn/scikit-learn/pull/5491#discussion_r43575167"},"pull_request":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5491"}}},"pull_request":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5491","id":48214039,"html_url":"https://github.com/scikit-learn/scikit-learn/pull/5491","diff_url":"https://github.com/scikit-learn/scikit-learn/pull/5491.diff","patch_url":"https://github.com/scikit-learn/scikit-learn/pull/5491.patch","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5491","number":5491,"state":"open","locked":false,"title":"[WIP] Gaussian Process-based hyper-parameter optimization","user":{"login":"fabianp","id":277639,"avatar_url":"https://avatars.githubusercontent.com/u/277639?v=3","gravatar_id":"","url":"https://api.github.com/users/fabianp","html_url":"https://github.com/fabianp","followers_url":"https://api.github.com/users/fabianp/followers","following_url":"https://api.github.com/users/fabianp/following{/other_user}","gists_url":"https://api.github.com/users/fabianp/gists{/gist_id}","starred_url":"https://api.github.com/users/fabianp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fabianp/subscriptions","organizations_url":"https://api.github.com/users/fabianp/orgs","repos_url":"https://api.github.com/users/fabianp/repos","events_url":"https://api.github.com/users/fabianp/events{/privacy}","received_events_url":"https://api.github.com/users/fabianp/received_events","type":"User","site_admin":false},"body":"Based on @sds-dubois code (PR #5185), this code adds an API that is more compatible with the GridSearchCV interface and some tests.\r\n\r\nThings still TODO are\r\n\r\n- [x] Make the example simpler and faster.\r\n- [x] Fix the failing test (```test_n_iter_smaller_n_iter```).\r\n- [ ] Add this object to the narrative documentation.\r\n- [x] Clearly state the way to specify hyperparameters. In particular, it seems fine with not adding the type of the params, i.e. both {'foo_param': [1, 2, 3]} and {'foo_param': ['int', [2, 3]]}. I'm not against it, but is should be clearly stated how hyperparameters need to be specified, what is accepted and what is not.\r\n- [x] Improve test coverage.\r\n- [x] ```GPSearchCV().scores_``` should have the same structure as ```GridSearchCV().grid_scores_```\r\n- [x] Random seed for ```_sample_candidates```\r\n- [x] Allow parallelization across CV folds.\r\n\r\nI think the module name is not ideal, since in the future we migth have other *SearchCV objects (e.g. based on trees). Any suggestions?\r\n\r\nCC @Djabbz ","created_at":"2015-10-20T15:10:25Z","updated_at":"2015-10-31T19:14:29Z","closed_at":null,"merged_at":null,"merge_commit_sha":"81ebac4cdf8c2c4a1f9343c69872cd22cdc4820b","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5491/commits","review_comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5491/comments","review_comment_url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/comments{/number}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5491/comments","statuses_url":"https://api.github.com/repos/scikit-learn/scikit-learn/statuses/ae685b88f21426d8b8d63fea97c1a6a30635781e","head":{"label":"fabianp:gp_hyperopt","ref":"gp_hyperopt","sha":"ae685b88f21426d8b8d63fea97c1a6a30635781e","user":{"login":"fabianp","id":277639,"avatar_url":"https://avatars.githubusercontent.com/u/277639?v=3","gravatar_id":"","url":"https://api.github.com/users/fabianp","html_url":"https://github.com/fabianp","followers_url":"https://api.github.com/users/fabianp/followers","following_url":"https://api.github.com/users/fabianp/following{/other_user}","gists_url":"https://api.github.com/users/fabianp/gists{/gist_id}","starred_url":"https://api.github.com/users/fabianp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fabianp/subscriptions","organizations_url":"https://api.github.com/users/fabianp/orgs","repos_url":"https://api.github.com/users/fabianp/repos","events_url":"https://api.github.com/users/fabianp/events{/privacy}","received_events_url":"https://api.github.com/users/fabianp/received_events","type":"User","site_admin":false},"repo":{"id":843235,"name":"scikit-learn","full_name":"fabianp/scikit-learn","owner":{"login":"fabianp","id":277639,"avatar_url":"https://avatars.githubusercontent.com/u/277639?v=3","gravatar_id":"","url":"https://api.github.com/users/fabianp","html_url":"https://github.com/fabianp","followers_url":"https://api.github.com/users/fabianp/followers","following_url":"https://api.github.com/users/fabianp/following{/other_user}","gists_url":"https://api.github.com/users/fabianp/gists{/gist_id}","starred_url":"https://api.github.com/users/fabianp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fabianp/subscriptions","organizations_url":"https://api.github.com/users/fabianp/orgs","repos_url":"https://api.github.com/users/fabianp/repos","events_url":"https://api.github.com/users/fabianp/events{/privacy}","received_events_url":"https://api.github.com/users/fabianp/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/fabianp/scikit-learn","description":"Machine Learning in Python","fork":true,"url":"https://api.github.com/repos/fabianp/scikit-learn","forks_url":"https://api.github.com/repos/fabianp/scikit-learn/forks","keys_url":"https://api.github.com/repos/fabianp/scikit-learn/keys{/key_id}","collaborators_url":"https://api.github.com/repos/fabianp/scikit-learn/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/fabianp/scikit-learn/teams","hooks_url":"https://api.github.com/repos/fabianp/scikit-learn/hooks","issue_events_url":"https://api.github.com/repos/fabianp/scikit-learn/issues/events{/number}","events_url":"https://api.github.com/repos/fabianp/scikit-learn/events","assignees_url":"https://api.github.com/repos/fabianp/scikit-learn/assignees{/user}","branches_url":"https://api.github.com/repos/fabianp/scikit-learn/branches{/branch}","tags_url":"https://api.github.com/repos/fabianp/scikit-learn/tags","blobs_url":"https://api.github.com/repos/fabianp/scikit-learn/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/fabianp/scikit-learn/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/fabianp/scikit-learn/git/refs{/sha}","trees_url":"https://api.github.com/repos/fabianp/scikit-learn/git/trees{/sha}","statuses_url":"https://api.github.com/repos/fabianp/scikit-learn/statuses/{sha}","languages_url":"https://api.github.com/repos/fabianp/scikit-learn/languages","stargazers_url":"https://api.github.com/repos/fabianp/scikit-learn/stargazers","contributors_url":"https://api.github.com/repos/fabianp/scikit-learn/contributors","subscribers_url":"https://api.github.com/repos/fabianp/scikit-learn/subscribers","subscription_url":"https://api.github.com/repos/fabianp/scikit-learn/subscription","commits_url":"https://api.github.com/repos/fabianp/scikit-learn/commits{/sha}","git_commits_url":"https://api.github.com/repos/fabianp/scikit-learn/git/commits{/sha}","comments_url":"https://api.github.com/repos/fabianp/scikit-learn/comments{/number}","issue_comment_url":"https://api.github.com/repos/fabianp/scikit-learn/issues/comments{/number}","contents_url":"https://api.github.com/repos/fabianp/scikit-learn/contents/{+path}","compare_url":"https://api.github.com/repos/fabianp/scikit-learn/compare/{base}...{head}","merges_url":"https://api.github.com/repos/fabianp/scikit-learn/merges","archive_url":"https://api.github.com/repos/fabianp/scikit-learn/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/fabianp/scikit-learn/downloads","issues_url":"https://api.github.com/repos/fabianp/scikit-learn/issues{/number}","pulls_url":"https://api.github.com/repos/fabianp/scikit-learn/pulls{/number}","milestones_url":"https://api.github.com/repos/fabianp/scikit-learn/milestones{/number}","notifications_url":"https://api.github.com/repos/fabianp/scikit-learn/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/fabianp/scikit-learn/labels{/name}","releases_url":"https://api.github.com/repos/fabianp/scikit-learn/releases{/id}","created_at":"2010-08-17T09:51:17Z","updated_at":"2015-10-19T11:51:34Z","pushed_at":"2015-10-30T12:33:06Z","git_url":"git://github.com/fabianp/scikit-learn.git","ssh_url":"git@github.com:fabianp/scikit-learn.git","clone_url":"https://github.com/fabianp/scikit-learn.git","svn_url":"https://github.com/fabianp/scikit-learn","homepage":"http://scikit-learn.org","size":68943,"stargazers_count":8,"watchers_count":8,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":8,"default_branch":"master"}},"base":{"label":"scikit-learn:master","ref":"master","sha":"6541f3f0fe04e7352965a10a86d46873bd28f2a9","user":{"login":"scikit-learn","id":365630,"avatar_url":"https://avatars.githubusercontent.com/u/365630?v=3","gravatar_id":"","url":"https://api.github.com/users/scikit-learn","html_url":"https://github.com/scikit-learn","followers_url":"https://api.github.com/users/scikit-learn/followers","following_url":"https://api.github.com/users/scikit-learn/following{/other_user}","gists_url":"https://api.github.com/users/scikit-learn/gists{/gist_id}","starred_url":"https://api.github.com/users/scikit-learn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scikit-learn/subscriptions","organizations_url":"https://api.github.com/users/scikit-learn/orgs","repos_url":"https://api.github.com/users/scikit-learn/repos","events_url":"https://api.github.com/users/scikit-learn/events{/privacy}","received_events_url":"https://api.github.com/users/scikit-learn/received_events","type":"Organization","site_admin":false},"repo":{"id":843222,"name":"scikit-learn","full_name":"scikit-learn/scikit-learn","owner":{"login":"scikit-learn","id":365630,"avatar_url":"https://avatars.githubusercontent.com/u/365630?v=3","gravatar_id":"","url":"https://api.github.com/users/scikit-learn","html_url":"https://github.com/scikit-learn","followers_url":"https://api.github.com/users/scikit-learn/followers","following_url":"https://api.github.com/users/scikit-learn/following{/other_user}","gists_url":"https://api.github.com/users/scikit-learn/gists{/gist_id}","starred_url":"https://api.github.com/users/scikit-learn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scikit-learn/subscriptions","organizations_url":"https://api.github.com/users/scikit-learn/orgs","repos_url":"https://api.github.com/users/scikit-learn/repos","events_url":"https://api.github.com/users/scikit-learn/events{/privacy}","received_events_url":"https://api.github.com/users/scikit-learn/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/scikit-learn/scikit-learn","description":"scikit-learn: machine learning in Python","fork":false,"url":"https://api.github.com/repos/scikit-learn/scikit-learn","forks_url":"https://api.github.com/repos/scikit-learn/scikit-learn/forks","keys_url":"https://api.github.com/repos/scikit-learn/scikit-learn/keys{/key_id}","collaborators_url":"https://api.github.com/repos/scikit-learn/scikit-learn/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/scikit-learn/scikit-learn/teams","hooks_url":"https://api.github.com/repos/scikit-learn/scikit-learn/hooks","issue_events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/events{/number}","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/events","assignees_url":"https://api.github.com/repos/scikit-learn/scikit-learn/assignees{/user}","branches_url":"https://api.github.com/repos/scikit-learn/scikit-learn/branches{/branch}","tags_url":"https://api.github.com/repos/scikit-learn/scikit-learn/tags","blobs_url":"https://api.github.com/repos/scikit-learn/scikit-learn/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/scikit-learn/scikit-learn/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/scikit-learn/scikit-learn/git/refs{/sha}","trees_url":"https://api.github.com/repos/scikit-learn/scikit-learn/git/trees{/sha}","statuses_url":"https://api.github.com/repos/scikit-learn/scikit-learn/statuses/{sha}","languages_url":"https://api.github.com/repos/scikit-learn/scikit-learn/languages","stargazers_url":"https://api.github.com/repos/scikit-learn/scikit-learn/stargazers","contributors_url":"https://api.github.com/repos/scikit-learn/scikit-learn/contributors","subscribers_url":"https://api.github.com/repos/scikit-learn/scikit-learn/subscribers","subscription_url":"https://api.github.com/repos/scikit-learn/scikit-learn/subscription","commits_url":"https://api.github.com/repos/scikit-learn/scikit-learn/commits{/sha}","git_commits_url":"https://api.github.com/repos/scikit-learn/scikit-learn/git/commits{/sha}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/comments{/number}","issue_comment_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments{/number}","contents_url":"https://api.github.com/repos/scikit-learn/scikit-learn/contents/{+path}","compare_url":"https://api.github.com/repos/scikit-learn/scikit-learn/compare/{base}...{head}","merges_url":"https://api.github.com/repos/scikit-learn/scikit-learn/merges","archive_url":"https://api.github.com/repos/scikit-learn/scikit-learn/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/scikit-learn/scikit-learn/downloads","issues_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues{/number}","pulls_url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls{/number}","milestones_url":"https://api.github.com/repos/scikit-learn/scikit-learn/milestones{/number}","notifications_url":"https://api.github.com/repos/scikit-learn/scikit-learn/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/labels{/name}","releases_url":"https://api.github.com/repos/scikit-learn/scikit-learn/releases{/id}","created_at":"2010-08-17T09:43:38Z","updated_at":"2015-10-31T14:45:30Z","pushed_at":"2015-10-31T18:11:15Z","git_url":"git://github.com/scikit-learn/scikit-learn.git","ssh_url":"git@github.com:scikit-learn/scikit-learn.git","clone_url":"https://github.com/scikit-learn/scikit-learn.git","svn_url":"https://github.com/scikit-learn/scikit-learn","homepage":"http://scikit-learn.org","size":140896,"stargazers_count":8063,"watchers_count":8063,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":4801,"mirror_url":null,"open_issues_count":806,"forks":4801,"open_issues":806,"watchers":8063,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5491"},"html":{"href":"https://github.com/scikit-learn/scikit-learn/pull/5491"},"issue":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5491"},"comments":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5491/comments"},"review_comments":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5491/comments"},"review_comment":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5491/commits"},"statuses":{"href":"https://api.github.com/repos/scikit-learn/scikit-learn/statuses/ae685b88f21426d8b8d63fea97c1a6a30635781e"}}}}