{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7884","labels_url":"https://api.github.com/repos/apache/spark/issues/7884/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7884/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7884/events","html_url":"https://github.com/apache/spark/pull/7884","id":98670248,"number":7884,"title":"[SPARK-7685][ML] Apply weights to different samples in Logistic Regression","user":{"login":"dbtsai","id":1134574,"avatar_url":"https://avatars.githubusercontent.com/u/1134574?v=3","gravatar_id":"","url":"https://api.github.com/users/dbtsai","html_url":"https://github.com/dbtsai","followers_url":"https://api.github.com/users/dbtsai/followers","following_url":"https://api.github.com/users/dbtsai/following{/other_user}","gists_url":"https://api.github.com/users/dbtsai/gists{/gist_id}","starred_url":"https://api.github.com/users/dbtsai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dbtsai/subscriptions","organizations_url":"https://api.github.com/users/dbtsai/orgs","repos_url":"https://api.github.com/users/dbtsai/repos","events_url":"https://api.github.com/users/dbtsai/events{/privacy}","received_events_url":"https://api.github.com/users/dbtsai/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":30,"created_at":"2015-08-03T04:40:54Z","updated_at":"2015-08-04T07:16:01Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7884","html_url":"https://github.com/apache/spark/pull/7884","diff_url":"https://github.com/apache/spark/pull/7884.diff","patch_url":"https://github.com/apache/spark/pull/7884.patch"},"body":"In fraud detection dataset, almost all the samples are negative while only couple of them are positive. This type of high imbalanced data will bias the models toward negative resulting poor performance. In python-scikit, they provide a correction allowing users to Over-/undersample the samples of each class according to the given weights. In auto mode, selects weights inversely proportional to class frequencies in the training set. This can be done in a more efficient way by multiplying the weights into loss and gradient instead of doing actual over/undersampling in the training dataset which is very expensive.\r\nhttp://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html\r\nOn the other hand, some of the training data maybe more important like the training samples from tenure users while the training samples from new users maybe less important. We should be able to provide another \"weight: Double\" information in the LabeledPoint to weight them differently in the learning algorithm.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/127505014","html_url":"https://github.com/apache/spark/pull/7884#issuecomment-127505014","issue_url":"https://api.github.com/repos/apache/spark/issues/7884","id":127505014,"user":{"login":"dbtsai","id":1134574,"avatar_url":"https://avatars.githubusercontent.com/u/1134574?v=3","gravatar_id":"","url":"https://api.github.com/users/dbtsai","html_url":"https://github.com/dbtsai","followers_url":"https://api.github.com/users/dbtsai/followers","following_url":"https://api.github.com/users/dbtsai/following{/other_user}","gists_url":"https://api.github.com/users/dbtsai/gists{/gist_id}","starred_url":"https://api.github.com/users/dbtsai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dbtsai/subscriptions","organizations_url":"https://api.github.com/users/dbtsai/orgs","repos_url":"https://api.github.com/users/dbtsai/repos","events_url":"https://api.github.com/users/dbtsai/events{/privacy}","received_events_url":"https://api.github.com/users/dbtsai/received_events","type":"User","site_admin":false},"created_at":"2015-08-04T07:16:01Z","updated_at":"2015-08-04T07:16:01Z","body":"This is very useful for us as well. Our training samples will have exponential decay weight as time goes by. I am trying to use it now, and it seems work well. Will be nice to have similar feature in GBDT which is used in our current production. Open a JIRA, and I can work on GBDT version."}}