{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/7258","labels_url":"https://api.github.com/repos/apache/spark/issues/7258/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/7258/comments","events_url":"https://api.github.com/repos/apache/spark/issues/7258/events","html_url":"https://github.com/apache/spark/pull/7258","id":93467012,"number":7258,"title":"[SPARK-6487][MLlib] Add sequential pattern mining algorithm PrefixSpan to Spark MLlib","user":{"login":"zhangjiajin","id":13159256,"avatar_url":"https://avatars.githubusercontent.com/u/13159256?v=3","gravatar_id":"","url":"https://api.github.com/users/zhangjiajin","html_url":"https://github.com/zhangjiajin","followers_url":"https://api.github.com/users/zhangjiajin/followers","following_url":"https://api.github.com/users/zhangjiajin/following{/other_user}","gists_url":"https://api.github.com/users/zhangjiajin/gists{/gist_id}","starred_url":"https://api.github.com/users/zhangjiajin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhangjiajin/subscriptions","organizations_url":"https://api.github.com/users/zhangjiajin/orgs","repos_url":"https://api.github.com/users/zhangjiajin/repos","events_url":"https://api.github.com/users/zhangjiajin/events{/privacy}","received_events_url":"https://api.github.com/users/zhangjiajin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":46,"created_at":"2015-07-07T08:06:15Z","updated_at":"2015-07-09T15:50:42Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/7258","html_url":"https://github.com/apache/spark/pull/7258","diff_url":"https://github.com/apache/spark/pull/7258.diff","patch_url":"https://github.com/apache/spark/pull/7258.patch"},"body":"Add parallel PrefixSpan algorithm and test file.\r\nSupport non-temporal sequences."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/120045016","html_url":"https://github.com/apache/spark/pull/7258#issuecomment-120045016","issue_url":"https://api.github.com/repos/apache/spark/issues/7258","id":120045016,"user":{"login":"mengxr","id":829644,"avatar_url":"https://avatars.githubusercontent.com/u/829644?v=3","gravatar_id":"","url":"https://api.github.com/users/mengxr","html_url":"https://github.com/mengxr","followers_url":"https://api.github.com/users/mengxr/followers","following_url":"https://api.github.com/users/mengxr/following{/other_user}","gists_url":"https://api.github.com/users/mengxr/gists{/gist_id}","starred_url":"https://api.github.com/users/mengxr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mengxr/subscriptions","organizations_url":"https://api.github.com/users/mengxr/orgs","repos_url":"https://api.github.com/users/mengxr/repos","events_url":"https://api.github.com/users/mengxr/events{/privacy}","received_events_url":"https://api.github.com/users/mengxr/received_events","type":"User","site_admin":false},"created_at":"2015-07-09T15:50:42Z","updated_at":"2015-07-09T15:50:42Z","body":"@zhangjiajin Since you already collected the frequent items (length-1 patterns) to driver, you don't need to keep the RDD of length-1 patterns. When generating the final patterns, recomputing the RDD is more expensive than parallelizing the collected ones.\r\n\r\nAnother comment is to separate local computation from the distributed ones. It makes the implementation easier to read. We can create a private object called `LocalPrefixSpan`, with `run(sequences: Array[Array[Int]], minCount: Int): Array[(Array[Int], Int)]`, then put all local methods under this object."}}