{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/6213","labels_url":"https://api.github.com/repos/apache/spark/issues/6213/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/6213/comments","events_url":"https://api.github.com/repos/apache/spark/issues/6213/events","html_url":"https://github.com/apache/spark/pull/6213","id":77170901,"number":6213,"title":"[WIP][MLLIB][SPARK-4675][SPARK-4823]RowSimilarity","user":{"login":"debasish83","id":510896,"avatar_url":"https://avatars.githubusercontent.com/u/510896?v=3","gravatar_id":"","url":"https://api.github.com/users/debasish83","html_url":"https://github.com/debasish83","followers_url":"https://api.github.com/users/debasish83/followers","following_url":"https://api.github.com/users/debasish83/following{/other_user}","gists_url":"https://api.github.com/users/debasish83/gists{/gist_id}","starred_url":"https://api.github.com/users/debasish83/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/debasish83/subscriptions","organizations_url":"https://api.github.com/users/debasish83/orgs","repos_url":"https://api.github.com/users/debasish83/repos","events_url":"https://api.github.com/users/debasish83/events{/privacy}","received_events_url":"https://api.github.com/users/debasish83/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":32,"created_at":"2015-05-17T00:42:35Z","updated_at":"2015-05-23T19:15:46Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/6213","html_url":"https://github.com/apache/spark/pull/6213","diff_url":"https://github.com/apache/spark/pull/6213.diff","patch_url":"https://github.com/apache/spark/pull/6213.patch"},"body":"@mengxr @srowen @rezazadeh\r\n\r\nFor RowMatrix with 100K columns, colSimilarity with bruteforce/dimsum sampling is used. This PR adds rowSimilarity to IndexedRowMatrix which outputs a CoordinateMatrix. For matrices where columns are > 1M, rowSimilarity flow scales better compared to column similarity flow.\r\n\r\nFor most applications, topK similar items requirement is much less than all available items and therefore the rowSimilarity API takes topK and threshold as input. topK and threshold help in improving shuffle space.\r\n\r\nFor MatrixFactorization model generally the columns for both user and product factors are ~50-200 and therefore the column similarity flow does not work for such cases. This PR also adds batch similarUsers and similarProducts (SPARK-4675).\r\n\r\nThe following ideas are added:\r\n\r\n1. Similarity computation is abstracted as Kernel\r\n2. Kernel implementations for Cosine, RBF, Euclidean and Product (for distributed matrix multiply) are added\r\n3. Tests cover Cosine Kernel. More tests will be added for Euclidean, RBF and Product kernels.\r\n4. IndexedRowMatrix object adds a kernalized distributed matrix multiply which is used by similarity computation.\r\n5. In examples, MovieLensSimilarity is added that shows col and row based flows on MovieLens as runtime experiment.\r\n6. Level-1 BLAS is used so that kernel abstraction can be used. We can either design the Kernel abstraction with Level-3 BLAS (might be difficult) or use BlockMatrix for distributed matrix multiply.\r\n\r\nNext steps:\r\n\r\n1. In MovieLensSimilarity add ALS + similarItems example\r\n2. Use RBF similarity in power iteration clustering flow\r\n\r\nFrom internal experiments, we have run 6M users, 1.6M items, 351M ratings through row similarity flow with topK=200 in 1.1 hr with 240 cores running over 30 nodes. We had difficult time in scaling column similarity flow since the topK optimization can't be added until reduce phase is done in that flow.\r\n\r\nOn MovieLens-1M and Netflix dataset I will report row and col similarity runtime comparisons."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/104937269","html_url":"https://github.com/apache/spark/pull/6213#issuecomment-104937269","issue_url":"https://api.github.com/repos/apache/spark/issues/6213","id":104937269,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-05-23T19:15:46Z","updated_at":"2015-05-23T19:15:46Z","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/33419/\nTest FAILed."}}