{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5104","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5104/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5104/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5104/events","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5104","id":100052147,"number":5104,"title":"[MRG+2] partial_fit for StandardScaler, MinMaxScaler and MaxAbsScaler","user":{"login":"giorgiop","id":2871319,"avatar_url":"https://avatars.githubusercontent.com/u/2871319?v=3","gravatar_id":"","url":"https://api.github.com/users/giorgiop","html_url":"https://github.com/giorgiop","followers_url":"https://api.github.com/users/giorgiop/followers","following_url":"https://api.github.com/users/giorgiop/following{/other_user}","gists_url":"https://api.github.com/users/giorgiop/gists{/gist_id}","starred_url":"https://api.github.com/users/giorgiop/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/giorgiop/subscriptions","organizations_url":"https://api.github.com/users/giorgiop/orgs","repos_url":"https://api.github.com/users/giorgiop/repos","events_url":"https://api.github.com/users/giorgiop/events{/privacy}","received_events_url":"https://api.github.com/users/giorgiop/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":28,"created_at":"2015-08-10T12:31:26Z","updated_at":"2015-09-16T17:15:43Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5104","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5104","diff_url":"https://github.com/scikit-learn/scikit-learn/pull/5104.diff","patch_url":"https://github.com/scikit-learn/scikit-learn/pull/5104.patch"},"body":"I am fixing #5028. This is only about `StandardScaler`. I moved the call to _handle_zeros_in_scale into `trasform` and `inverse_trasform`. This is necessary since otherwise we lose the true value of std (and var) that is needed for incremental computation —or otherwise we would need an additional variable only for that, which is not very elegant. Incidentally, this addresses #4609. I have also slightly extended `utils.extmath._batch_mean_variance_update` and did some refactoring in `decomposition.incremental_PCA`, that calls the former. \r\n\r\nA more conceptual change is the following. `partial_fit` must interpret an 1d array as a row matrix, i.e. must be able to process batches of size 1. This clashes with the behaviour of `fit` which by default assumes that 1d input arrays are column matrices. I think this is a design issue that should be tackled more generally, hence out of scope of this PR. See for example #4511. @GaelVaroquaux may want to comment here.\r\n\r\nI have not written tests on the interplay of subsequent calls to `fit` and `partial_fit`, as the semantics does not seem defined yet. See #3896\r\n\r\nRelated open discussions:\r\n- scalers #2514\r\n- partial_fit #3907 #3896\r\n\r\nEDIT ~ to do list:\r\n- [x] `partial_fit` for `StandardScaler`, `MaxAbsScaler` and `MinMaxScaler`\r\n- [x] support to CSC and CSR for all but the last\r\n- [x] ~~as discussed, there is now a validator for 2d input, that interprets it as 1-row matrix (instead of 1-column) in case of need: `row_or_2d`~~ input is not checked any more, see below and #5152 \r\n- [x] `scale_` is now common attribute to all scalers, and it is different from the stat that it is computed from, e.g. `mean_`, `max_abs` etc. The latter comes before `_handle_zeros_in_scale`, the former after it. `partial_fit` requires the true stats value for the incremental update. As we need to save it, I also expose it for the object --also in the case only `fit` is used.\r\n- [x] cosmetics in docstring, functions calls and cython identations\r\n- [x] remove silent copy inside `_handle_zeros_in_scale`\r\n- [x] double check `incr_mean_variance_axis0` in `sparsefuncs_fast.pyx` for avoiding some redundant computation\r\n- [x] additional test for numerical stability of `_batch_mean_variance_update`\r\n- [x] respect attribute name convention\r\n- [x] test for numerical issues from Numpy < 1.9\r\n- [x] cosmetics for line breaks in expression etc.\r\n- [x] remove tests with 1d inputs"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/140809149","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5104#issuecomment-140809149","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5104","id":140809149,"user":{"login":"giorgiop","id":2871319,"avatar_url":"https://avatars.githubusercontent.com/u/2871319?v=3","gravatar_id":"","url":"https://api.github.com/users/giorgiop","html_url":"https://github.com/giorgiop","followers_url":"https://api.github.com/users/giorgiop/followers","following_url":"https://api.github.com/users/giorgiop/following{/other_user}","gists_url":"https://api.github.com/users/giorgiop/gists{/gist_id}","starred_url":"https://api.github.com/users/giorgiop/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/giorgiop/subscriptions","organizations_url":"https://api.github.com/users/giorgiop/orgs","repos_url":"https://api.github.com/users/giorgiop/repos","events_url":"https://api.github.com/users/giorgiop/events{/privacy}","received_events_url":"https://api.github.com/users/giorgiop/received_events","type":"User","site_admin":false},"created_at":"2015-09-16T17:15:43Z","updated_at":"2015-09-16T17:15:43Z","body":"Thank you again @MechCoder. I have started to work on issues related to 1d inputs. Scalers were not fully covered by #5152. I will need to fix some tests too indeed."}}