{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/3098","labels_url":"https://api.github.com/repos/apache/spark/issues/3098/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/3098/comments","events_url":"https://api.github.com/repos/apache/spark/issues/3098/events","html_url":"https://github.com/apache/spark/pull/3098","id":47789888,"number":3098,"title":"[MLLIB] SPARK-4231, SPARK-3066: Add RankingMetrics to examples.MovieLensALS","user":{"login":"debasish83","id":510896,"avatar_url":"https://avatars.githubusercontent.com/u/510896?v=3","gravatar_id":"","url":"https://api.github.com/users/debasish83","html_url":"https://github.com/debasish83","followers_url":"https://api.github.com/users/debasish83/followers","following_url":"https://api.github.com/users/debasish83/following{/other_user}","gists_url":"https://api.github.com/users/debasish83/gists{/gist_id}","starred_url":"https://api.github.com/users/debasish83/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/debasish83/subscriptions","organizations_url":"https://api.github.com/users/debasish83/orgs","repos_url":"https://api.github.com/users/debasish83/repos","events_url":"https://api.github.com/users/debasish83/events{/privacy}","received_events_url":"https://api.github.com/users/debasish83/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":49,"created_at":"2014-11-05T01:38:16Z","updated_at":"2015-04-01T00:22:02Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/3098","html_url":"https://github.com/apache/spark/pull/3098","diff_url":"https://github.com/apache/spark/pull/3098.diff","patch_url":"https://github.com/apache/spark/pull/3098.patch"},"body":"@mengxr @srowen\r\n\r\nTo validate ALS enhancements as proposed in https://issues.apache.org/jira/browse/SPARK-2426, RMSE along with the RankingMetrics measures are important to look at.  \r\n\r\nThis PR adds a flag --validateProducts to examples.MovieLensALS.\r\n\r\nDefault validateProducts is false and we compute RMSE and MAP for test set related to product recommendation.\r\n\r\n./bin/spark-submit --master spark://tusca09lmlvt00c.uswin.ad.vzwcorp.com:7077 --jars /Users/v606014/.m2/repository/com/github/scopt/scopt_2.10/3.2.0/scopt_2.10-3.2.0.jar --total-executor-cores 4 --executor-memory 4g --driver-memory 1g --class org.apache.spark.examples.mllib.MovieLensALS ./examples/target/spark-examples_2.10-1.2.0-SNAPSHOT.jar --kryo --lambda 0.065 hdfs://localhost:8020/sandbox/movielens/\r\n2014-11-04 17:15:24.262 java[4568:1903] Unable to load realm mapping info from SCDynamicStore\r\n14/11/04 17:15:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\nGot 1000209 ratings from 6040 users on 3706 movies.\r\nTraining: 799926, test: 200283.\r\nTest RMSE = 0.8965005871008247 MAP = 7.438473265235346.\r\n\r\n--validateProducts will validate user recommendation for each product\r\n\r\n./bin/spark-submit --master spark://tusca09lmlvt00c.uswin.ad.vzwcorp.com:7077 --jars /Users/v606014/.m2/repository/com/github/scopt/scopt_2.10/3.2.0/scopt_2.10-3.2.0.jar --total-executor-cores 4 --executor-memory 4g --driver-memory 1g --class org.apache.spark.examples.mllib.MovieLensALS ./examples/target/spark-examples_2.10-1.2.0-SNAPSHOT.jar --kryo --lambda 0.065 --validateProducts hdfs://localhost:8020/sandbox/movielens/\r\n2014-11-04 17:16:18.652 java[4635:1903] Unable to load realm mapping info from SCDynamicStore\r\n14/11/04 17:16:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\nGot 1000209 ratings from 6040 users on 3706 movies.\r\nTraining: 800014, test: 200195.\r\nTest RMSE = 0.8986539583457682 MAP = 12.243775391575324.\r\n\r\nSean,\r\n\r\nAre we looking at the right numbers here ? MAP for Movielens dataset is around 12.243. You did similar experiments for oryx/myrrix before...\r\n\r\nWe can perhaps make the test set generation more intelligent but I went with random sampling for now since I was looking at MAP measure..\r\n\r\nFor prec@k I am not sure what's the right k number to choose at...I do a sweep over k to choose sweet spot internally.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/88292172","html_url":"https://github.com/apache/spark/pull/3098#issuecomment-88292172","issue_url":"https://api.github.com/repos/apache/spark/issues/3098","id":88292172,"user":{"login":"debasish83","id":510896,"avatar_url":"https://avatars.githubusercontent.com/u/510896?v=3","gravatar_id":"","url":"https://api.github.com/users/debasish83","html_url":"https://github.com/debasish83","followers_url":"https://api.github.com/users/debasish83/followers","following_url":"https://api.github.com/users/debasish83/following{/other_user}","gists_url":"https://api.github.com/users/debasish83/gists{/gist_id}","starred_url":"https://api.github.com/users/debasish83/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/debasish83/subscriptions","organizations_url":"https://api.github.com/users/debasish83/orgs","repos_url":"https://api.github.com/users/debasish83/repos","events_url":"https://api.github.com/users/debasish83/events{/privacy}","received_events_url":"https://api.github.com/users/debasish83/received_events","type":"User","site_admin":false},"created_at":"2015-04-01T00:22:02Z","updated_at":"2015-04-01T00:22:02Z","body":"If we move computeRankingMetrics and computeRMSE to a better place, I can guard it through tests..."}}