{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/27707375","id":27707375,"diff_hunk":"@@ -138,14 +141,122 @@ class MatrixFactorizationModel(\n   }\n \n   private def recommend(\n-      recommendToFeatures: Array[Double],\n-      recommendableFeatures: RDD[(Int, Array[Double])],\n-      num: Int): Array[(Int, Double)] = {\n-    val scored = recommendableFeatures.map { case (id,features) =>\n-      (id, blas.ddot(features.length, recommendToFeatures, 1, features, 1))\n+    recommendToFeatures: Array[Double],\n+    recommendableFeatures: RDD[(Int, Array[Double])],\n+    num: Int): Array[(Int, Double)] = {\n+    val recommendToVector = Vectors.dense(recommendToFeatures)\n+    val scored = recommendableFeatures.map {\n+      case (id, features) =>\n+        (id, BLAS.dot(recommendToVector, Vectors.dense(features)))\n     }\n     scored.top(num)(Ordering.by(_._2))\n   }\n+\n+  /**\n+   * Recommends topK products for all users\n+   *\n+   * @param num how many products to return for every user.\n+   * @return [(Int, Array[Rating])] objects, where every tuple contains a userID and an array of\n+   * rating objects which contains the same userId, recommended productID and a \"score\" in the\n+   * rating field. Semantics of score is same as recommendProducts API\n+   */\n+  def recommendProductsForUsers(num: Int): RDD[(Int, Array[Rating])] = {\n+    val topK = userFeatures.map { x => (x._1, num) }\n+    recommendProductsForUsers(topK)\n+  }\n+\n+  /**\n+   * Recommends topK users for all products\n+   *\n+   * @param num how many users to return for every product.\n+   * @return [(Int, Array[Rating])] objects, where every tuple contains a productID and an array\n+   * of rating objects which contains the recommended userId, same productID and a \"score\" in the\n+   * rating field. Semantics of score is same as recommendUsers API\n+   */\n+  def recommendUsersForProducts(num: Int): RDD[(Int, Array[Rating])] = {\n+    val topK = productFeatures.map { x => (x._1, num) }\n+    recommendUsersForProducts(topK)\n+  }\n+\n+  val ord = Ordering.by[Rating, Double](x => x.rating)\n+  case class FeatureTopK(feature: Vector, topK: Int)","path":"mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala","position":106,"original_position":106,"commit_id":"3a0c4eb7f81ee0845f4945d395f6652c965f941b","original_commit_id":"3a0c4eb7f81ee0845f4945d395f6652c965f941b","user":{"login":"mengxr","id":829644,"avatar_url":"https://avatars.githubusercontent.com/u/829644?v=3","gravatar_id":"","url":"https://api.github.com/users/mengxr","html_url":"https://github.com/mengxr","followers_url":"https://api.github.com/users/mengxr/followers","following_url":"https://api.github.com/users/mengxr/following{/other_user}","gists_url":"https://api.github.com/users/mengxr/gists{/gist_id}","starred_url":"https://api.github.com/users/mengxr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mengxr/subscriptions","organizations_url":"https://api.github.com/users/mengxr/orgs","repos_url":"https://api.github.com/users/mengxr/repos","events_url":"https://api.github.com/users/mengxr/events{/privacy}","received_events_url":"https://api.github.com/users/mengxr/received_events","type":"User","site_admin":false},"body":"This is not necessary if we use a global `num`.","created_at":"2015-04-02T23:47:59Z","updated_at":"2015-04-02T23:47:59Z","html_url":"https://github.com/apache/spark/pull/3098#discussion_r27707375","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/3098","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/27707375"},"html":{"href":"https://github.com/apache/spark/pull/3098#discussion_r27707375"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/3098"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/3098","id":23878737,"html_url":"https://github.com/apache/spark/pull/3098","diff_url":"https://github.com/apache/spark/pull/3098.diff","patch_url":"https://github.com/apache/spark/pull/3098.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/3098","number":3098,"state":"open","locked":false,"title":"[MLLIB] SPARK-4231, SPARK-3066: Add RankingMetrics to examples.MovieLensALS","user":{"login":"debasish83","id":510896,"avatar_url":"https://avatars.githubusercontent.com/u/510896?v=3","gravatar_id":"","url":"https://api.github.com/users/debasish83","html_url":"https://github.com/debasish83","followers_url":"https://api.github.com/users/debasish83/followers","following_url":"https://api.github.com/users/debasish83/following{/other_user}","gists_url":"https://api.github.com/users/debasish83/gists{/gist_id}","starred_url":"https://api.github.com/users/debasish83/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/debasish83/subscriptions","organizations_url":"https://api.github.com/users/debasish83/orgs","repos_url":"https://api.github.com/users/debasish83/repos","events_url":"https://api.github.com/users/debasish83/events{/privacy}","received_events_url":"https://api.github.com/users/debasish83/received_events","type":"User","site_admin":false},"body":"@mengxr @srowen\r\n\r\nTo validate ALS enhancements as proposed in https://issues.apache.org/jira/browse/SPARK-2426, RMSE along with the RankingMetrics measures are important to look at.  \r\n\r\nThis PR adds a flag --validateProducts to examples.MovieLensALS.\r\n\r\nDefault validateProducts is false and we compute RMSE and MAP for test set related to product recommendation.\r\n\r\n./bin/spark-submit --master spark://tusca09lmlvt00c.uswin.ad.vzwcorp.com:7077 --jars /Users/v606014/.m2/repository/com/github/scopt/scopt_2.10/3.2.0/scopt_2.10-3.2.0.jar --total-executor-cores 4 --executor-memory 4g --driver-memory 1g --class org.apache.spark.examples.mllib.MovieLensALS ./examples/target/spark-examples_2.10-1.2.0-SNAPSHOT.jar --kryo --lambda 0.065 hdfs://localhost:8020/sandbox/movielens/\r\n2014-11-04 17:15:24.262 java[4568:1903] Unable to load realm mapping info from SCDynamicStore\r\n14/11/04 17:15:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\nGot 1000209 ratings from 6040 users on 3706 movies.\r\nTraining: 799926, test: 200283.\r\nTest RMSE = 0.8965005871008247 MAP = 7.438473265235346.\r\n\r\n--validateProducts will validate user recommendation for each product\r\n\r\n./bin/spark-submit --master spark://tusca09lmlvt00c.uswin.ad.vzwcorp.com:7077 --jars /Users/v606014/.m2/repository/com/github/scopt/scopt_2.10/3.2.0/scopt_2.10-3.2.0.jar --total-executor-cores 4 --executor-memory 4g --driver-memory 1g --class org.apache.spark.examples.mllib.MovieLensALS ./examples/target/spark-examples_2.10-1.2.0-SNAPSHOT.jar --kryo --lambda 0.065 --validateProducts hdfs://localhost:8020/sandbox/movielens/\r\n2014-11-04 17:16:18.652 java[4635:1903] Unable to load realm mapping info from SCDynamicStore\r\n14/11/04 17:16:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\nGot 1000209 ratings from 6040 users on 3706 movies.\r\nTraining: 800014, test: 200195.\r\nTest RMSE = 0.8986539583457682 MAP = 12.243775391575324.\r\n\r\nSean,\r\n\r\nAre we looking at the right numbers here ? MAP for Movielens dataset is around 12.243. You did similar experiments for oryx/myrrix before...\r\n\r\nWe can perhaps make the test set generation more intelligent but I went with random sampling for now since I was looking at MAP measure..\r\n\r\nFor prec@k I am not sure what's the right k number to choose at...I do a sweep over k to choose sweet spot internally.\r\n","created_at":"2014-11-05T01:38:16Z","updated_at":"2015-04-02T23:47:59Z","closed_at":null,"merged_at":null,"merge_commit_sha":"594e452b02f043439d73c4eedaf55c21b5192953","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/3098/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/3098/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/3098/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/3a0c4eb7f81ee0845f4945d395f6652c965f941b","head":{"label":"debasish83:irmetrics","ref":"irmetrics","sha":"3a0c4eb7f81ee0845f4945d395f6652c965f941b","user":{"login":"debasish83","id":510896,"avatar_url":"https://avatars.githubusercontent.com/u/510896?v=3","gravatar_id":"","url":"https://api.github.com/users/debasish83","html_url":"https://github.com/debasish83","followers_url":"https://api.github.com/users/debasish83/followers","following_url":"https://api.github.com/users/debasish83/following{/other_user}","gists_url":"https://api.github.com/users/debasish83/gists{/gist_id}","starred_url":"https://api.github.com/users/debasish83/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/debasish83/subscriptions","organizations_url":"https://api.github.com/users/debasish83/orgs","repos_url":"https://api.github.com/users/debasish83/repos","events_url":"https://api.github.com/users/debasish83/events{/privacy}","received_events_url":"https://api.github.com/users/debasish83/received_events","type":"User","site_admin":false},"repo":{"id":17971510,"name":"spark","full_name":"debasish83/spark","owner":{"login":"debasish83","id":510896,"avatar_url":"https://avatars.githubusercontent.com/u/510896?v=3","gravatar_id":"","url":"https://api.github.com/users/debasish83","html_url":"https://github.com/debasish83","followers_url":"https://api.github.com/users/debasish83/followers","following_url":"https://api.github.com/users/debasish83/following{/other_user}","gists_url":"https://api.github.com/users/debasish83/gists{/gist_id}","starred_url":"https://api.github.com/users/debasish83/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/debasish83/subscriptions","organizations_url":"https://api.github.com/users/debasish83/orgs","repos_url":"https://api.github.com/users/debasish83/repos","events_url":"https://api.github.com/users/debasish83/events{/privacy}","received_events_url":"https://api.github.com/users/debasish83/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/debasish83/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/debasish83/spark","forks_url":"https://api.github.com/repos/debasish83/spark/forks","keys_url":"https://api.github.com/repos/debasish83/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/debasish83/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/debasish83/spark/teams","hooks_url":"https://api.github.com/repos/debasish83/spark/hooks","issue_events_url":"https://api.github.com/repos/debasish83/spark/issues/events{/number}","events_url":"https://api.github.com/repos/debasish83/spark/events","assignees_url":"https://api.github.com/repos/debasish83/spark/assignees{/user}","branches_url":"https://api.github.com/repos/debasish83/spark/branches{/branch}","tags_url":"https://api.github.com/repos/debasish83/spark/tags","blobs_url":"https://api.github.com/repos/debasish83/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/debasish83/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/debasish83/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/debasish83/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/debasish83/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/debasish83/spark/languages","stargazers_url":"https://api.github.com/repos/debasish83/spark/stargazers","contributors_url":"https://api.github.com/repos/debasish83/spark/contributors","subscribers_url":"https://api.github.com/repos/debasish83/spark/subscribers","subscription_url":"https://api.github.com/repos/debasish83/spark/subscription","commits_url":"https://api.github.com/repos/debasish83/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/debasish83/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/debasish83/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/debasish83/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/debasish83/spark/contents/{+path}","compare_url":"https://api.github.com/repos/debasish83/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/debasish83/spark/merges","archive_url":"https://api.github.com/repos/debasish83/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/debasish83/spark/downloads","issues_url":"https://api.github.com/repos/debasish83/spark/issues{/number}","pulls_url":"https://api.github.com/repos/debasish83/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/debasish83/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/debasish83/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/debasish83/spark/labels{/name}","releases_url":"https://api.github.com/repos/debasish83/spark/releases{/id}","created_at":"2014-03-21T07:18:33Z","updated_at":"2015-04-01T03:45:16Z","pushed_at":"2015-04-01T06:57:43Z","git_url":"git://github.com/debasish83/spark.git","ssh_url":"git@github.com:debasish83/spark.git","clone_url":"https://github.com/debasish83/spark.git","svn_url":"https://github.com/debasish83/spark","homepage":null,"size":93113,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"305abe1e57450f49e3ec4dffb073c5adf17cadef","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","created_at":"2014-02-25T08:00:08Z","updated_at":"2015-04-02T23:47:18Z","pushed_at":"2015-04-02T23:47:30Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":1464404,"stargazers_count":3469,"watchers_count":3469,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":2988,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":318,"forks":2988,"open_issues":318,"watchers":3469,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/3098"},"html":{"href":"https://github.com/apache/spark/pull/3098"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/3098"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/3098/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/3098/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/3098/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/3a0c4eb7f81ee0845f4945d395f6652c965f941b"}}}}