{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4460","labels_url":"https://api.github.com/repos/apache/spark/issues/4460/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4460/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4460/events","html_url":"https://github.com/apache/spark/pull/4460","id":56970437,"number":4460,"title":"SPARK-4588 [MLLIB] [WIP] Add API for feature attributes","user":{"login":"srowen","id":822522,"avatar_url":"https://avatars.githubusercontent.com/u/822522?v=3","gravatar_id":"","url":"https://api.github.com/users/srowen","html_url":"https://github.com/srowen","followers_url":"https://api.github.com/users/srowen/followers","following_url":"https://api.github.com/users/srowen/following{/other_user}","gists_url":"https://api.github.com/users/srowen/gists{/gist_id}","starred_url":"https://api.github.com/users/srowen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/srowen/subscriptions","organizations_url":"https://api.github.com/users/srowen/orgs","repos_url":"https://api.github.com/users/srowen/repos","events_url":"https://api.github.com/users/srowen/events{/privacy}","received_events_url":"https://api.github.com/users/srowen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2015-02-08T22:14:30Z","updated_at":"2015-02-20T11:22:39Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4460","html_url":"https://github.com/apache/spark/pull/4460","diff_url":"https://github.com/apache/spark/pull/4460.diff","patch_url":"https://github.com/apache/spark/pull/4460.patch"},"body":"@mengxr\r\n\r\nThis is an early checkin to see if this is in the right direction at all.\r\n\r\n- Should this have a counterpart \"builder\" class, or setters that make a new copy of the object?\r\n- In the JIRA you mention feature dimension. Dumb question, but is that just the cardinality of categorical features?\r\n- I structured it all with maps, and I wonder if that's OK"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/75223695","html_url":"https://github.com/apache/spark/pull/4460#issuecomment-75223695","issue_url":"https://api.github.com/repos/apache/spark/issues/4460","id":75223695,"user":{"login":"mengxr","id":829644,"avatar_url":"https://avatars.githubusercontent.com/u/829644?v=3","gravatar_id":"","url":"https://api.github.com/users/mengxr","html_url":"https://github.com/mengxr","followers_url":"https://api.github.com/users/mengxr/followers","following_url":"https://api.github.com/users/mengxr/following{/other_user}","gists_url":"https://api.github.com/users/mengxr/gists{/gist_id}","starred_url":"https://api.github.com/users/mengxr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mengxr/subscriptions","organizations_url":"https://api.github.com/users/mengxr/orgs","repos_url":"https://api.github.com/users/mengxr/repos","events_url":"https://api.github.com/users/mengxr/events{/privacy}","received_events_url":"https://api.github.com/users/mengxr/received_events","type":"User","site_admin":false},"created_at":"2015-02-20T11:22:39Z","updated_at":"2015-02-20T11:22:39Z","body":"@srowen If we mark a string column categorical, it may be hard to answer how many categories it has without looking at the data. If a column is marked categorical, it should store the information of categories in the metadata and store only integer/double in the column. The difference is that we cannot merge a string column with a double column into a vector column without turning the string column into an integer/double column. So I feel that we should treat string column and categorical column separately.\r\n\r\nFor the type hierarchy, if we add a type, let's discuss what algorithms would actually use it:\r\n\r\n1. numeric (slightly broader than continuous): linear algorithms can take them, while decision tree needs to bin them first (if the number of distinct values are large)\r\n2. categorical: decision tree can take them, while linear algorithms needs dummy coding.\r\n3. binary: both linear algorithms and decision tree can take them directly.\r\n4. discrete/ordinal: ?\r\n\r\nMaybe for the base `Attribute` trait, we can have `cardinality: Int`. For continuous data, we put `inf` or `-1`, for ordinal data, we put the number of distinct values. Let's try to write down the API and see how it looks.\r\n\r\nNot a concern at this stage, for `AttributeGroup` we might need to handle the storage efficiently to avoid GC pressure. It may come out with millions of features, and we will hit GC if we use too many objects to store the attributes. Well, this is maybe too early to worry about."}}