{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4270","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4270/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4270/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4270/events","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4270","id":58260770,"number":4270,"title":"[WIP] Gaussian process revisited","user":{"login":"jmetzen","id":1116263,"avatar_url":"https://avatars.githubusercontent.com/u/1116263?v=3","gravatar_id":"","url":"https://api.github.com/users/jmetzen","html_url":"https://github.com/jmetzen","followers_url":"https://api.github.com/users/jmetzen/followers","following_url":"https://api.github.com/users/jmetzen/following{/other_user}","gists_url":"https://api.github.com/users/jmetzen/gists{/gist_id}","starred_url":"https://api.github.com/users/jmetzen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmetzen/subscriptions","organizations_url":"https://api.github.com/users/jmetzen/orgs","repos_url":"https://api.github.com/users/jmetzen/repos","events_url":"https://api.github.com/users/jmetzen/events{/privacy}","received_events_url":"https://api.github.com/users/jmetzen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/milestones/9","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/milestones/9/labels","id":190845,"number":9,"title":"1.0","description":"This milestone is not really associated with a deadline.\r\nI intend it as a list of things we think have to be done before we get there.","creator":{"login":"amueller","id":449558,"avatar_url":"https://avatars.githubusercontent.com/u/449558?v=3","gravatar_id":"","url":"https://api.github.com/users/amueller","html_url":"https://github.com/amueller","followers_url":"https://api.github.com/users/amueller/followers","following_url":"https://api.github.com/users/amueller/following{/other_user}","gists_url":"https://api.github.com/users/amueller/gists{/gist_id}","starred_url":"https://api.github.com/users/amueller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amueller/subscriptions","organizations_url":"https://api.github.com/users/amueller/orgs","repos_url":"https://api.github.com/users/amueller/repos","events_url":"https://api.github.com/users/amueller/events{/privacy}","received_events_url":"https://api.github.com/users/amueller/received_events","type":"User","site_admin":false},"open_issues":12,"closed_issues":18,"state":"open","created_at":"2012-10-07T13:00:07Z","updated_at":"2015-02-23T00:46:06Z","due_on":"2015-12-31T05:00:00Z","closed_at":null},"comments":22,"created_at":"2015-02-19T20:03:01Z","updated_at":"2015-02-24T17:47:59Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/4270","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4270","diff_url":"https://github.com/scikit-learn/scikit-learn/pull/4270.diff","patch_url":"https://github.com/scikit-learn/scikit-learn/pull/4270.patch"},"body":"This PR intents to address some of the points on the wish list for Gaussian processes that have been discussed on the mailing list recently.  Among the features are\r\n * kernel engineering (combining basic kernels using \"+\" and \"*\" operators)\r\n * gradient-based hyperparameter optimization\r\n * estimating the global noise level as part of the hyperparameter optimization (using a \"WhiteKernel\")\r\n * Gaussian process classification\r\n * sampling of functions from GP prior and GP posterior\r\n\r\nExamples for this PR are (for the moment) in examples/gaussian_process/new. \r\n\r\nThis PR deliberately avoids any interdependency with the existing GP code and starts completely from scratch. This is surely debatable but in my opinion the interface of the existing GP is part of the problem as it separates the specification of the kernel and its parameters which makes more complex kernels close to impossible to specify. Moreover, the existing code is not amenable for extensions, in my opinion. \r\n\r\nThis PR is still in an early state with several open issues and incomplete documentation. Feedback is very welcome. There are definitely some open questions left in this PR, in particular:\r\n * how could we make a transition from the \"old\" GP code to the new one? Do we want to do that or should this code live rather as an extension module outside of sklearn? So far, I haven't removed any of the old code.\r\n * clarify the details of specifying advanced kernels, in particular the lower and upper boundary for the hyperparameters. If someone has an alternative idea to the current triple-based notation (preliminary documentation of this in kernel.py), I am very open for changing it.\r\n\r\nList of todos:\r\n  - [ ] explicit basis function for GPR (GPML book, chapter 2.7)\r\n  - [ ] support multiple restarts of gradient-based optimization\r\n  - [ ] optimize all hyperparameters with specified search-space, keep other hyperparameters fixed (currently only all or none is supported) \r\n  - [ ] replace deepcopy(kernel) by kernel_with_params()\r\n  - [ ] test kernels for stationarity\r\n  - [ ] implement special method __eq__ for GPs\r\n  - [ ] check dimensionality of returned value of GPC.predict_proba()\r\n  - [ ] complete documentation\r\n\r\nSome plots:\r\nGP classification\r\n![gpc](https://cloud.githubusercontent.com/assets/1116263/6274705/bde22c48-b879-11e4-9522-dfce288ccba8.png)\r\n\r\nEstimation of noise-level:\r\n![noise-estimation](https://cloud.githubusercontent.com/assets/1116263/6274724/d4dc711a-b879-11e4-9582-43eaaa3dfed2.png)\r\n\r\nIllustration of GP prior and GP posterior based on sample-function:\r\n![prior_posterior](https://cloud.githubusercontent.com/assets/1116263/6274736/eb01f2b2-b879-11e4-9b68-4976be198cd7.png)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/75807678","html_url":"https://github.com/scikit-learn/scikit-learn/pull/4270#issuecomment-75807678","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4270","id":75807678,"user":{"login":"jmetzen","id":1116263,"avatar_url":"https://avatars.githubusercontent.com/u/1116263?v=3","gravatar_id":"","url":"https://api.github.com/users/jmetzen","html_url":"https://github.com/jmetzen","followers_url":"https://api.github.com/users/jmetzen/followers","following_url":"https://api.github.com/users/jmetzen/following{/other_user}","gists_url":"https://api.github.com/users/jmetzen/gists{/gist_id}","starred_url":"https://api.github.com/users/jmetzen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmetzen/subscriptions","organizations_url":"https://api.github.com/users/jmetzen/orgs","repos_url":"https://api.github.com/users/jmetzen/repos","events_url":"https://api.github.com/users/jmetzen/events{/privacy}","received_events_url":"https://api.github.com/users/jmetzen/received_events","type":"User","site_admin":false},"created_at":"2015-02-24T17:47:59Z","updated_at":"2015-02-24T17:47:59Z","body":"> I had forgotten the potential for high complexity after kernel composition. So yes, bounds belong to kernel. In that case, instead of the tuple, maybe something like alpha=value, alpha_bounds=(lower, upper) could be an option? As far as I understand, in the options you provide with scalar/pair/triple, there is never the possibility only to specify one bound, right? So alpha=None, alpha_bounds=(lower, upper) would correspond to passing (lower, upper), alpha=value, alpha_bounds=(lower, upper) would be (lower, value, upper) and alpha=value, alpha_bounds=None would correspond to alpha.\r\n> \r\n> It adds readability. I wonder if it adds too much to write, though.\r\n\r\nThe issue with the \"alpha=value, alpha_bounds=(lower, upper)\" way of specifying the box constraints (which was also my first try at it) is that it does not extend to specifying a ConstantKernel in the \"literal\" form like `kernel = (0.1, 1.0, 10.0) * RBF(1.0)`. Using the pair/triple notation was my best idea for specifying a box constraint on a ConstantKernel in literal form (without having to write explicitly `ConstantKernel(1.0, (0.1, 10.0)) * RBF(1.0)`). If we use the pair/triple notation there, we can as well use it in all other places,. Otherwise, I would also prefer the alpha, alpha_bounds way in `__init__`. "}}