{"action":"created","issue":{"url":"https://api.github.com/repos/UWFlow/rmc/issues/254","labels_url":"https://api.github.com/repos/UWFlow/rmc/issues/254/labels{/name}","comments_url":"https://api.github.com/repos/UWFlow/rmc/issues/254/comments","events_url":"https://api.github.com/repos/UWFlow/rmc/issues/254/events","html_url":"https://github.com/UWFlow/rmc/pull/254","id":120706845,"number":254,"title":"Prototype: Course recommendation engine using Apache Spark","user":{"login":"ccqi","id":6456601,"avatar_url":"https://avatars.githubusercontent.com/u/6456601?v=3","gravatar_id":"","url":"https://api.github.com/users/ccqi","html_url":"https://github.com/ccqi","followers_url":"https://api.github.com/users/ccqi/followers","following_url":"https://api.github.com/users/ccqi/following{/other_user}","gists_url":"https://api.github.com/users/ccqi/gists{/gist_id}","starred_url":"https://api.github.com/users/ccqi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ccqi/subscriptions","organizations_url":"https://api.github.com/users/ccqi/orgs","repos_url":"https://api.github.com/users/ccqi/repos","events_url":"https://api.github.com/users/ccqi/events{/privacy}","received_events_url":"https://api.github.com/users/ccqi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-12-07T06:21:34Z","updated_at":"2015-12-09T06:01:43Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/UWFlow/rmc/pulls/254","html_url":"https://github.com/UWFlow/rmc/pull/254","diff_url":"https://github.com/UWFlow/rmc/pull/254.diff","patch_url":"https://github.com/UWFlow/rmc/pull/254.patch"},"body":"I have been thinking of this idea for a while, but never had the time to implement it until now. \r\n\r\n## Proposal\r\nSince UW Flow is approaching 12000 users and growing, I think this a good time to setup a course recommendation engine.\r\n\r\nAfter some research, I believe [Apache Spark](http://spark.apache.org/docs/latest/index.html) is a great tool to use for this purpose since it offers fast, parallelized cluster computing as well as a lot of rich tools and libraries for machine learning algorithms.\r\n\r\nI also want to use [Collaborative Filtering](http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html) as the algorithm for the recommendation engine. The advantages of this algorithm is that it takes account of similar user's preferences to make recommendations.\r\n\r\n## Prototype Explanation\r\nThe prototype I made sends a list of triples (user_id, course_id, rating) to spark. The rating, for now, is our *Liked It* button. Spark uses the data to train a model using [ALS] (http://bugra.github.io/work/notes/2014-04-19/alternating-least-squares-method-for-collaborative-filtering) (Alternating least squares). The model is trained to *predict* the course rating (0.0-1.0) of all courses not yet rated by the user. Then given a user and the trained model, it returns the top 5 highest courses rated by the user as *recommendations*.\r\n\r\n## Running this PR\r\nIf you want to test this PR, you need to [install and build](https://spark.apache.org/downloads.html) spark and then set $SPARK_HOME to your spark install directory (uncomment some code in engine.py to set this up). Also install these python modules:\r\n```\r\npip install numpy\r\npip install py4j\r\n```\r\nThen call \r\n\r\n```make recommendations user_id=<user_id>```\r\n\r\n* user_id is the m.User.id property of each user\r\n* user_id parameter is optional, if not provided, the code will recommend courses to a random user in the database\r\n\r\n## Next Steps\r\nIf we decide to go with this approach, here are some next steps and problems I think we need to solve:\r\n\r\n**Optimization**\r\nIn order to optimize the engine, we should run metrics against UW flow's real user data. (I am currently running the algorithm with some test data that I generated)\r\n\r\nThe most common step to [evaluate](http://machinelearningmastery.com/how-to-evaluate-machine-learning-algorithms/) learning algorithms is to split the available data into 3 sets: training (60%), validation (20%) and test (20%). Then we choose the optimal parameters based on the lowest error on the cross validation set. Finally, we measure the final error of the algorithm on the test set.\r\n\r\nTo do this, I need access to the entire or at least a subset of our user data.\r\n\r\nAnother possible optimization is to train the algorithm using more data (eg. shortlisted courses, easiness, usefulness) \r\n\r\n**Integration**\r\nIf we go with the approach, I suggest we integrate the engine with the application like this:\r\n* The model is *trained* every day with the new user data - run it with aggregator.py? \r\n* Whenever /user is loaded, front-end queries the recommender for the top \"n\" recommendations\r\n* Display the top \"n\" results to the user in the front-end, maybe we can reuse this design \r\n![image](https://cloud.githubusercontent.com/assets/6456601/11619926/42a44016-9c7b-11e5-913b-4cce9e16d9a1.png)\r\n\r\n**Deployment**\r\nWe need to set-up spark for [deployment](http://spark.apache.org/docs/latest/cluster-overview.html).\r\n\r\nThe biggest advantage to using machine learning is that since our data is infinitely growing, our recommendations should also get much better over time. "},"comment":{"url":"https://api.github.com/repos/UWFlow/rmc/issues/comments/163119894","html_url":"https://github.com/UWFlow/rmc/pull/254#issuecomment-163119894","issue_url":"https://api.github.com/repos/UWFlow/rmc/issues/254","id":163119894,"user":{"login":"JGulbronson","id":2644780,"avatar_url":"https://avatars.githubusercontent.com/u/2644780?v=3","gravatar_id":"","url":"https://api.github.com/users/JGulbronson","html_url":"https://github.com/JGulbronson","followers_url":"https://api.github.com/users/JGulbronson/followers","following_url":"https://api.github.com/users/JGulbronson/following{/other_user}","gists_url":"https://api.github.com/users/JGulbronson/gists{/gist_id}","starred_url":"https://api.github.com/users/JGulbronson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JGulbronson/subscriptions","organizations_url":"https://api.github.com/users/JGulbronson/orgs","repos_url":"https://api.github.com/users/JGulbronson/repos","events_url":"https://api.github.com/users/JGulbronson/events{/privacy}","received_events_url":"https://api.github.com/users/JGulbronson/received_events","type":"User","site_admin":false},"created_at":"2015-12-09T06:01:43Z","updated_at":"2015-12-09T06:01:43Z","body":"So I'm trying to test this locally, and am running in to an issue. (Note, I have production data on my machine, so it's actually training with real data)\r\n\r\nI got spark by running `brew install apache-spark`, and am running your script with `PYTHONPATH=.. pyspark data/engine.py userId` where 'userId' is the id for my local profile.\r\n\r\nHowever, it's throwing an error, saying `ERROR:__main__:User has not rated any courses yet` after it says the model has been trained. I've definitely rated courses, so any idea what's happening here?\r\n\r\nI really love the idea, and it's great you put together a prototype PR so we can play around with it before putting up a formal one :)\r\n\r\nLet me know when you get the chance, I want to get this working!"}}