{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4270","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4270/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4270/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4270/events","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4270","id":58260770,"number":4270,"title":"[WIP] Gaussian process revisited","user":{"login":"jmetzen","id":1116263,"avatar_url":"https://avatars.githubusercontent.com/u/1116263?v=3","gravatar_id":"","url":"https://api.github.com/users/jmetzen","html_url":"https://github.com/jmetzen","followers_url":"https://api.github.com/users/jmetzen/followers","following_url":"https://api.github.com/users/jmetzen/following{/other_user}","gists_url":"https://api.github.com/users/jmetzen/gists{/gist_id}","starred_url":"https://api.github.com/users/jmetzen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmetzen/subscriptions","organizations_url":"https://api.github.com/users/jmetzen/orgs","repos_url":"https://api.github.com/users/jmetzen/repos","events_url":"https://api.github.com/users/jmetzen/events{/privacy}","received_events_url":"https://api.github.com/users/jmetzen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/milestones/9","html_url":"https://github.com/scikit-learn/scikit-learn/milestones/1.0","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/milestones/9/labels","id":190845,"number":9,"title":"1.0","description":"This milestone is not really associated with a deadline.\r\nI intend it as a list of things we think have to be done before we get there.","creator":{"login":"amueller","id":449558,"avatar_url":"https://avatars.githubusercontent.com/u/449558?v=3","gravatar_id":"","url":"https://api.github.com/users/amueller","html_url":"https://github.com/amueller","followers_url":"https://api.github.com/users/amueller/followers","following_url":"https://api.github.com/users/amueller/following{/other_user}","gists_url":"https://api.github.com/users/amueller/gists{/gist_id}","starred_url":"https://api.github.com/users/amueller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amueller/subscriptions","organizations_url":"https://api.github.com/users/amueller/orgs","repos_url":"https://api.github.com/users/amueller/repos","events_url":"https://api.github.com/users/amueller/events{/privacy}","received_events_url":"https://api.github.com/users/amueller/received_events","type":"User","site_admin":false},"open_issues":11,"closed_issues":19,"state":"open","created_at":"2012-10-07T13:00:07Z","updated_at":"2015-03-05T18:29:29Z","due_on":"2015-12-31T05:00:00Z","closed_at":null},"comments":34,"created_at":"2015-02-19T20:03:01Z","updated_at":"2015-03-08T19:29:14Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/4270","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4270","diff_url":"https://github.com/scikit-learn/scikit-learn/pull/4270.diff","patch_url":"https://github.com/scikit-learn/scikit-learn/pull/4270.patch"},"body":"This PR intents to address some of the points on the wish list for Gaussian processes that have been discussed on the mailing list recently.  Among the features are\r\n * kernel engineering (combining basic kernels using \"+\" and \"*\" operators)\r\n * gradient-based hyperparameter optimization\r\n * estimating the global noise level as part of the hyperparameter optimization (using a \"WhiteKernel\")\r\n * Gaussian process classification\r\n * sampling of functions from GP prior and GP posterior\r\n\r\nExamples for this PR are (for the moment) in examples/gaussian_process/new. \r\n\r\nThis PR deliberately avoids any interdependency with the existing GP code and starts completely from scratch. This is surely debatable but in my opinion the interface of the existing GP is part of the problem as it separates the specification of the kernel and its parameters which makes more complex kernels close to impossible to specify. Moreover, the existing code is not amenable for extensions, in my opinion. \r\n\r\nThis PR is still in an early state with several open issues and incomplete documentation. Feedback is very welcome. There are definitely some open questions left in this PR, in particular:\r\n * how could we make a transition from the \"old\" GP code to the new one? Do we want to do that or should this code live rather as an extension module outside of sklearn? So far, I haven't removed any of the old code.\r\n * clarify the details of specifying advanced kernels, in particular the lower and upper boundary for the hyperparameters. If someone has an alternative idea to the current triple-based notation (preliminary documentation of this in kernel.py), I am very open for changing it.\r\n\r\nList of todos:\r\n  - [ ] explicit basis function for GPR (GPML book, chapter 2.7)\r\n  - [ ] support multiple restarts of gradient-based optimization\r\n  - [ ] optimize all hyperparameters with specified search-space, keep other hyperparameters fixed (currently only all or none is supported) \r\n  - [x] replace deepcopy(kernel) by clone_with_theta()\r\n  - [x] test kernels for stationarity\r\n  - [x] implement special method `__eq__` for GP kernels\r\n  - [ ] check dimensionality of returned value of GPC.predict_proba()\r\n  - [ ] test GP kernels with kernel regression\r\n  - [ ] deprecate legacy GP code (GaussianProcess, correlation_models, regression_models)\r\n  - [ ] adapt legacy tests/examples if possible; remove otherwise \r\n  - [ ] move CO2 data to mldata\r\n  - [ ] complete documentation\r\n\r\nSome plots:\r\nGP classification\r\n![gpc](https://cloud.githubusercontent.com/assets/1116263/6274705/bde22c48-b879-11e4-9522-dfce288ccba8.png)\r\n\r\nEstimation of noise-level:\r\n![noise-estimation](https://cloud.githubusercontent.com/assets/1116263/6274724/d4dc711a-b879-11e4-9582-43eaaa3dfed2.png)\r\n\r\nIllustration of GP prior and GP posterior based on sample-function:\r\n![prior_posterior](https://cloud.githubusercontent.com/assets/1116263/6274736/eb01f2b2-b879-11e4-9b68-4976be198cd7.png)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/77769250","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4270#issuecomment-77769250","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4270","id":77769250,"user":{"login":"eickenberg","id":1306635,"avatar_url":"https://avatars.githubusercontent.com/u/1306635?v=3","gravatar_id":"","url":"https://api.github.com/users/eickenberg","html_url":"https://github.com/eickenberg","followers_url":"https://api.github.com/users/eickenberg/followers","following_url":"https://api.github.com/users/eickenberg/following{/other_user}","gists_url":"https://api.github.com/users/eickenberg/gists{/gist_id}","starred_url":"https://api.github.com/users/eickenberg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eickenberg/subscriptions","organizations_url":"https://api.github.com/users/eickenberg/orgs","repos_url":"https://api.github.com/users/eickenberg/repos","events_url":"https://api.github.com/users/eickenberg/events{/privacy}","received_events_url":"https://api.github.com/users/eickenberg/received_events","type":"User","site_admin":false},"created_at":"2015-03-08T19:29:14Z","updated_at":"2015-03-08T19:29:14Z","body":"Thanks for the effort!! This looks like it is really going in the right direction and I hope we can make it converge soon :)\r\n\r\nI have a few comments that I will place in the code. Essentially, the points that I find need some extra discussion are:\r\n- The kernel parameters are always called `theta` and are always in one variable despite the fact that you went to the length of bring the `get_params` meta-magic into the abstract class. I understand that you do this in order to group the variables to be optimized on, but you could store that information elsewhere and accept different parameter names. Calling a parameter `sigma` in the docstring and `theta` in the signature of `__init__` is extremely bewildering ...\r\n- The shorthand `(.5, 0., 1.) * other_kernel == ConstantKernel(.5, 0., 1) * other_kernel` as a generalization of `.5 * other_kernel == ConstantKernel(.5) * other_kernel` still bothers me. It is unconventional and will surpise users. I don't see why `C(.5, 0., 1.) * other_kernel` with `ConstantKernel as C` is not enough.\r\n\r\nAfter this is addressed, I think we should be good to go. What do the others say? @amueller @mblondel @kastnerkyle @jakevdp "}}