{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5251","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5251/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5251/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5251/events","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5251","id":105999677,"number":5251,"title":"[MRG+1] add multinomial SAG solver for LogisticRegression","user":{"login":"TomDLT","id":11065596,"avatar_url":"https://avatars.githubusercontent.com/u/11065596?v=3","gravatar_id":"","url":"https://api.github.com/users/TomDLT","html_url":"https://github.com/TomDLT","followers_url":"https://api.github.com/users/TomDLT/followers","following_url":"https://api.github.com/users/TomDLT/following{/other_user}","gists_url":"https://api.github.com/users/TomDLT/gists{/gist_id}","starred_url":"https://api.github.com/users/TomDLT/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TomDLT/subscriptions","organizations_url":"https://api.github.com/users/TomDLT/orgs","repos_url":"https://api.github.com/users/TomDLT/repos","events_url":"https://api.github.com/users/TomDLT/events{/privacy}","received_events_url":"https://api.github.com/users/TomDLT/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2015-09-11T12:21:51Z","updated_at":"2015-11-19T22:50:13Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/pulls/5251","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5251","diff_url":"https://github.com/scikit-learn/scikit-learn/pull/5251.diff","patch_url":"https://github.com/scikit-learn/scikit-learn/pull/5251.patch"},"body":"TODO\r\n- [ ] remove new convergence warning in the tests\r\n\r\nI added multinomial sag solver in LogisticRegression.\r\nThe results are identical to lbfgs and newton-cg multinomial solvers.\r\n___\r\nThe benchmark [(gist)](https://gist.github.com/TomDLT/da7c44fa277003e7549e) on RCV1 dataset is very promising. I only took 4 classes of the dataset, and removed samples that have more than one classe.\r\n```python\r\nn_samples_train = 129073\r\nn_samples_test = 23149\r\nn_features = 47236\r\nn_classes = 4\r\n```\r\n![dloss](https://cloud.githubusercontent.com/assets/11065596/9814439/73b5f374-588e-11e5-94f4-adc82287026b.png)\r\n![train_score](https://cloud.githubusercontent.com/assets/11065596/9814441/76b4dd60-588e-11e5-8183-7a6fb733fc13.png)\r\n![test_score](https://cloud.githubusercontent.com/assets/11065596/9814440/7578b106-588e-11e5-874c-c56f4ac607f4.png)\r\n___\r\n\r\nI also added a `plot_logistic_multinomial.py` to show the difference between OvR and Multinomial.\r\n![ovr](https://cloud.githubusercontent.com/assets/11065596/9814355/8e521902-588d-11e5-8c2b-a962200e1e66.png)\r\n![multinomial](https://cloud.githubusercontent.com/assets/11065596/9814356/8f41eea0-588d-11e5-8f16-47bce1e89dce.png)\r\n___\r\nIn the same spirit, I could probably add a multinomial LogisticRegression in SGD (previously proposed in #849). Do we want it?"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/158225445","html_url":"https://github.com/scikit-learn/scikit-learn/pull/5251#issuecomment-158225445","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/5251","id":158225445,"user":{"login":"MechCoder","id":1867024,"avatar_url":"https://avatars.githubusercontent.com/u/1867024?v=3","gravatar_id":"","url":"https://api.github.com/users/MechCoder","html_url":"https://github.com/MechCoder","followers_url":"https://api.github.com/users/MechCoder/followers","following_url":"https://api.github.com/users/MechCoder/following{/other_user}","gists_url":"https://api.github.com/users/MechCoder/gists{/gist_id}","starred_url":"https://api.github.com/users/MechCoder/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MechCoder/subscriptions","organizations_url":"https://api.github.com/users/MechCoder/orgs","repos_url":"https://api.github.com/users/MechCoder/repos","events_url":"https://api.github.com/users/MechCoder/events{/privacy}","received_events_url":"https://api.github.com/users/MechCoder/received_events","type":"User","site_admin":false},"created_at":"2015-11-19T22:50:13Z","updated_at":"2015-11-19T22:50:13Z","body":"@TomDLT Sorry that I am looking at the sag code rather than the support for multinomial loss. But I have a very basic question. In the paper this is the implementation,\r\n\r\n    Kappa = 1\r\n    S_k = 0.0\r\n    for k in range(n_iter):\r\n         # Sample one in n_samples\r\n         # Note that Kappa and S_k are ever growing and not reset everywhere.\r\n         Kappa = Kappa * ( 1 -  lambda * step_size)\r\n         Update S_k = S_k-1 + step_size / (Kappa * num_seen)\r\n\r\n\r\n    # The implementation in sklearn.\r\n    # Effectively k is n_iter * sample_size\r\n    S = np.ones(n_iter)\r\n    for i in range(n_iter):\r\n        # Kappa being reset to 1.0\r\n        Kappa = 1.0\r\n        S[0] = step_size / (Kappa * num_seen)\r\n        for sample_iter in range(sample_size):\r\n            Kappa = Kappa * (1 - lambda * alpha)\r\n            if sample_iter != 0\r\n                S[sample_iter] = S[sample_(iter - 1)] +  step_size / (Kappa * num_seen)\r\n\r\n\r\nThe implementation in sklearn makes sense as well (if you treat every iteration across `n_samples` as a pass over new data but in that case `num_seen` should also be reset) but clearly it seems  both are not equivalent. That is the algorithm in the paper does not reset `Kappa`, ie `w_scale` or `cumulative_sums` anytime at all, (especially for everytime `max_iter % k` reaches n_samples .\r\n\r\nIs this a new implementation in sklearn?\r\ncc: @amueller "}}