{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/6742","labels_url":"https://api.github.com/repos/apache/spark/issues/6742/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/6742/comments","events_url":"https://api.github.com/repos/apache/spark/issues/6742/events","html_url":"https://github.com/apache/spark/pull/6742","id":86929809,"number":6742,"title":"[Spark-8169] [ML] Add StopWordsRemover as a transformer","user":{"login":"hhbyyh","id":7981698,"avatar_url":"https://avatars.githubusercontent.com/u/7981698?v=3","gravatar_id":"","url":"https://api.github.com/users/hhbyyh","html_url":"https://github.com/hhbyyh","followers_url":"https://api.github.com/users/hhbyyh/followers","following_url":"https://api.github.com/users/hhbyyh/following{/other_user}","gists_url":"https://api.github.com/users/hhbyyh/gists{/gist_id}","starred_url":"https://api.github.com/users/hhbyyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hhbyyh/subscriptions","organizations_url":"https://api.github.com/users/hhbyyh/orgs","repos_url":"https://api.github.com/users/hhbyyh/repos","events_url":"https://api.github.com/users/hhbyyh/events{/privacy}","received_events_url":"https://api.github.com/users/hhbyyh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-06-10T11:08:50Z","updated_at":"2015-07-31T02:52:58Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/6742","html_url":"https://github.com/apache/spark/pull/6742","diff_url":"https://github.com/apache/spark/pull/6742.diff","patch_url":"https://github.com/apache/spark/pull/6742.patch"},"body":"jira: https://issues.apache.org/jira/browse/SPARK-8169 \r\n\r\nstop words: http://en.wikipedia.org/wiki/Stop_words\r\n\r\nStopWordsRemover takes a string array column and outputs a string array column with all defined stop words removed. The transformer should also come with a standard set of stop words as default. \r\n\r\nCurrently I used a minimum stop words set since on some [case](http://nlp.stanford.edu/IR-book/html/htmledition/dropping-common-terms-stop-words-1.html), small set of stop words is preferred.\r\nASCII char has been tested, Yet I cannot check it in due to style check.\r\n\r\nFurther thought,\r\n1. Maybe I should use OpenHashSet. Is it recommended?\r\n2. Currently I leave the null in input array untouched, i.e. Array(null, null) => Array(null, null). \r\n3. If the current stop words set looks too limited, any suggestion for replacement? We can have something similar to the one in [SKlearn](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/stop_words.py). \r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/126549335","html_url":"https://github.com/apache/spark/pull/6742#issuecomment-126549335","issue_url":"https://api.github.com/repos/apache/spark/issues/6742","id":126549335,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-07-31T02:52:58Z","updated_at":"2015-07-31T02:52:58Z","body":"Merged build started."}}