{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7937","labels_url":"https://api.github.com/repos/apache/spark/issues/7937/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7937/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7937/events","html_url":"https://github.com/apache/spark/pull/7937","id":99016620,"number":7937,"title":"[SPARK-9540] [MLLIB] optimize PrefixSpan implementation","user":{"login":"mengxr","id":829644,"avatar_url":"https://avatars.githubusercontent.com/u/829644?v=3","gravatar_id":"","url":"https://api.github.com/users/mengxr","html_url":"https://github.com/mengxr","followers_url":"https://api.github.com/users/mengxr/followers","following_url":"https://api.github.com/users/mengxr/following{/other_user}","gists_url":"https://api.github.com/users/mengxr/gists{/gist_id}","starred_url":"https://api.github.com/users/mengxr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mengxr/subscriptions","organizations_url":"https://api.github.com/users/mengxr/orgs","repos_url":"https://api.github.com/users/mengxr/repos","events_url":"https://api.github.com/users/mengxr/events{/privacy}","received_events_url":"https://api.github.com/users/mengxr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":32,"created_at":"2015-08-04T17:16:37Z","updated_at":"2015-08-05T00:00:57Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7937","html_url":"https://github.com/apache/spark/pull/7937","diff_url":"https://github.com/apache/spark/pull/7937.diff","patch_url":"https://github.com/apache/spark/pull/7937.patch"},"body":"This is a major refactoring of the PrefixSpan implementation. It contains the following changes:\r\n\r\n1. Expand prefix with one item at a time. The existing implementation generates all subsets for each itemset, which might have scalability issue when the itemset is large.\r\n2. Use a new internal format. `<(12)(31)>` is represented by `[0, 1, 2, 0, 1, 3, 0]` internally. We use `0` because negative numbers are used to indicates partial prefix items, e.g., `_2` is represented by `-2`.\r\n3. Remember the start indices of all partial projections in the projected postfix to help next projection.\r\n4. Reuse the original sequence array for projected postfixes.\r\n5. Use `Prefix` IDs in aggregation rather than its content.\r\n6. Use `ArrayBuilder` for building primitive arrays.\r\n7. Expose `maxLocalProjDBSize`.\r\n8. Tests are not changed except using `0` instead of `-1` as the delimiter.\r\n\r\n`Postfix`'s API doc should be a good place to start.\r\n\r\nCloses #7594 \r\n\r\n@feynmanliang @zhangjiajin "},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/127799824","html_url":"https://github.com/apache/spark/pull/7937#issuecomment-127799824","issue_url":"https://api.github.com/repos/apache/spark/issues/7937","id":127799824,"user":{"login":"feynmanliang","id":990069,"avatar_url":"https://avatars.githubusercontent.com/u/990069?v=3","gravatar_id":"","url":"https://api.github.com/users/feynmanliang","html_url":"https://github.com/feynmanliang","followers_url":"https://api.github.com/users/feynmanliang/followers","following_url":"https://api.github.com/users/feynmanliang/following{/other_user}","gists_url":"https://api.github.com/users/feynmanliang/gists{/gist_id}","starred_url":"https://api.github.com/users/feynmanliang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/feynmanliang/subscriptions","organizations_url":"https://api.github.com/users/feynmanliang/orgs","repos_url":"https://api.github.com/users/feynmanliang/repos","events_url":"https://api.github.com/users/feynmanliang/events{/privacy}","received_events_url":"https://api.github.com/users/feynmanliang/received_events","type":"User","site_admin":false},"created_at":"2015-08-05T00:00:57Z","updated_at":"2015-08-05T00:00:57Z","body":"LGTM"}}