{"action":"created","issue":{"url":"https://api.github.com/repos/lensacom/sparkit-learn/issues/49","labels_url":"https://api.github.com/repos/lensacom/sparkit-learn/issues/49/labels{/name}","comments_url":"https://api.github.com/repos/lensacom/sparkit-learn/issues/49/comments","events_url":"https://api.github.com/repos/lensacom/sparkit-learn/issues/49/events","html_url":"https://github.com/lensacom/sparkit-learn/issues/49","id":93539775,"number":49,"title":"ValueError: This solver needs samples of at least 2 classes in the data","user":{"login":"mrshanth","id":5435145,"avatar_url":"https://avatars.githubusercontent.com/u/5435145?v=3","gravatar_id":"","url":"https://api.github.com/users/mrshanth","html_url":"https://github.com/mrshanth","followers_url":"https://api.github.com/users/mrshanth/followers","following_url":"https://api.github.com/users/mrshanth/following{/other_user}","gists_url":"https://api.github.com/users/mrshanth/gists{/gist_id}","starred_url":"https://api.github.com/users/mrshanth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrshanth/subscriptions","organizations_url":"https://api.github.com/users/mrshanth/orgs","repos_url":"https://api.github.com/users/mrshanth/repos","events_url":"https://api.github.com/users/mrshanth/events{/privacy}","received_events_url":"https://api.github.com/users/mrshanth/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/lensacom/sparkit-learn/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-07-07T14:10:03Z","updated_at":"2015-07-07T19:39:45Z","closed_at":null,"body":"Hi,\r\n\r\nI am using SparkLinearSVC. The code is as follows:\r\n```\r\nsvm_model = SparkLinearSVC(class_weight='auto')\r\nsvm_fitted = svm_model.fit(train_Z,classes=np.unique(train_y))\r\n```\r\nand I get the following error:\r\n```\r\nFile \"/DATA/sdw1/hadoop/yarn/local/usercache/ad79139/filecache/328/spark-assembly-1.2.1.2.2.4.2-2-hadoop2.6.0.2.2.4.2-2.jar/pyspark/worker.py\", line 98, in process\r\n    serializer.dump_stream(func(split_index, iterator), outfile)\r\n  File \"/usr/hdp/2.2.4.2-2/spark/python/pyspark/rdd.py\", line 2081, in pipeline_func\r\n    return func(split, prev_func(split, iterator))\r\n  File \"/usr/hdp/2.2.4.2-2/spark/python/pyspark/rdd.py\", line 2081, in pipeline_func\r\n    return func(split, prev_func(split, iterator))\r\n  File \"/usr/hdp/2.2.4.2-2/spark/python/pyspark/rdd.py\", line 258, in func\r\n    return f(iterator)\r\n  File \"/usr/hdp/2.2.4.2-2/spark/python/pyspark/rdd.py\", line 820, in <lambda>\r\n    return self.mapPartitions(lambda x: [sum(x)]).reduce(operator.add)\r\n  File \"/usr/lib/python2.6/site-packages/splearn/linear_model/base.py\", line 81, in <lambda>\r\n    mapper = lambda X_y: super(cls, self).fit(\r\n  File \"/usr/lib64/python2.6/site-packages/sklearn/svm/classes.py\", line 207, in fit\r\n    self.loss\r\n  File \"/usr/lib64/python2.6/site-packages/sklearn/svm/base.py\", line 809, in _fit_liblinear\r\n    \" class: %r\" % classes_[0])\r\nValueError: This solver needs samples of at least 2 classes in the data, but the data contains only one class: 0\r\n```\r\nwhereas, I have 2 classes, namely 0 and 1. The block size of the DictRDD is 2000. The percentage of classes 0 and 1 are 92% and 8% respectively"},"comment":{"url":"https://api.github.com/repos/lensacom/sparkit-learn/issues/comments/119314002","html_url":"https://github.com/lensacom/sparkit-learn/issues/49#issuecomment-119314002","issue_url":"https://api.github.com/repos/lensacom/sparkit-learn/issues/49","id":119314002,"user":{"login":"kszucs","id":961747,"avatar_url":"https://avatars.githubusercontent.com/u/961747?v=3","gravatar_id":"","url":"https://api.github.com/users/kszucs","html_url":"https://github.com/kszucs","followers_url":"https://api.github.com/users/kszucs/followers","following_url":"https://api.github.com/users/kszucs/following{/other_user}","gists_url":"https://api.github.com/users/kszucs/gists{/gist_id}","starred_url":"https://api.github.com/users/kszucs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kszucs/subscriptions","organizations_url":"https://api.github.com/users/kszucs/orgs","repos_url":"https://api.github.com/users/kszucs/repos","events_url":"https://api.github.com/users/kszucs/events{/privacy}","received_events_url":"https://api.github.com/users/kszucs/received_events","type":"User","site_admin":false},"created_at":"2015-07-07T19:39:45Z","updated_at":"2015-07-07T19:39:45Z","body":"Sadly this is a bug indeed. Sparkit trains sklearn's linear models in parallel, then averages them in a reduce step. There is at least one block, which contains only one of the labels. Try the following:\r\n```python\r\ntrain_Z[:, 'y'].map(lambda x: np.unique(x).size).filter(lambda x: x < 2).count()\r\n```\r\nYou could try to randomize the train data to avoid blocks with one label, but this is still waiting for a clever solution."}}