{"action":"created","issue":{"url":"https://api.github.com/repos/bigdatagenomics/eggo/issues/142","labels_url":"https://api.github.com/repos/bigdatagenomics/eggo/issues/142/labels{/name}","comments_url":"https://api.github.com/repos/bigdatagenomics/eggo/issues/142/comments","events_url":"https://api.github.com/repos/bigdatagenomics/eggo/issues/142/events","html_url":"https://github.com/bigdatagenomics/eggo/issues/142","id":117783538,"number":142,"title":"Upgrade Spark from 1.3 to 1.5 to make use of project Tungsten performance improvements. ","user":{"login":"NeillGibson","id":15608993,"avatar_url":"https://avatars.githubusercontent.com/u/15608993?v=3","gravatar_id":"","url":"https://api.github.com/users/NeillGibson","html_url":"https://github.com/NeillGibson","followers_url":"https://api.github.com/users/NeillGibson/followers","following_url":"https://api.github.com/users/NeillGibson/following{/other_user}","gists_url":"https://api.github.com/users/NeillGibson/gists{/gist_id}","starred_url":"https://api.github.com/users/NeillGibson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NeillGibson/subscriptions","organizations_url":"https://api.github.com/users/NeillGibson/orgs","repos_url":"https://api.github.com/users/NeillGibson/repos","events_url":"https://api.github.com/users/NeillGibson/events{/privacy}","received_events_url":"https://api.github.com/users/NeillGibson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-11-19T10:21:05Z","updated_at":"2015-11-22T14:16:54Z","closed_at":null,"body":"Hi, \r\n\r\nCould please modify the eggo cluster provision / configuration scripts to install / use a recent version of Apache Spark?\r\n\r\nWhen I currently start a cluster using Eggo and start the following shells\r\n\r\n/home/ec2-user/adam/bin/adam-shell\r\n/opt/cloudera/parcels/CDH-5.4.8-1.cdh5.4.8.p0.4/lib/spark/bin/spark-shell\r\n\r\nI see that the spark version is version 1.3.0 .\r\n\r\nOn the spark website I can see that there are spark releases for version 1.5.x \r\n\r\nhttps://spark.apache.org/news/spark-1-5-2-released.html \r\n\r\nThis project Tungsten video and slides mention that a lot of performance improvements have been added in the Spark 1.4 and 1.5 versions. \r\nhttps://spark-summit.org/2015/events/deep-dive-into-project-tungsten-bringing-spark-closer-to-bare-metal/\r\n\r\nI am currently doing some performance testing for our analytic queries and would like to make sure that I am making use of latest performance improving features. \r\n\r\nThank you very much! \r\n\r\nNeill\r\n"},"comment":{"url":"https://api.github.com/repos/bigdatagenomics/eggo/issues/comments/158762563","html_url":"https://github.com/bigdatagenomics/eggo/issues/142#issuecomment-158762563","issue_url":"https://api.github.com/repos/bigdatagenomics/eggo/issues/142","id":158762563,"user":{"login":"NeillGibson","id":15608993,"avatar_url":"https://avatars.githubusercontent.com/u/15608993?v=3","gravatar_id":"","url":"https://api.github.com/users/NeillGibson","html_url":"https://github.com/NeillGibson","followers_url":"https://api.github.com/users/NeillGibson/followers","following_url":"https://api.github.com/users/NeillGibson/following{/other_user}","gists_url":"https://api.github.com/users/NeillGibson/gists{/gist_id}","starred_url":"https://api.github.com/users/NeillGibson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NeillGibson/subscriptions","organizations_url":"https://api.github.com/users/NeillGibson/orgs","repos_url":"https://api.github.com/users/NeillGibson/repos","events_url":"https://api.github.com/users/NeillGibson/events{/privacy}","received_events_url":"https://api.github.com/users/NeillGibson/received_events","type":"User","site_admin":false},"created_at":"2015-11-22T14:16:54Z","updated_at":"2015-11-22T14:16:54Z","body":"Hi, \r\n\r\nCurrently Eggo installs cdh5.4.8. \r\n\r\nCdh 5.5.0 would install with Apache Spark 5.0.\r\nhttp://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_rn_new_in_550.html#concept_onz_f2h_mr_unique_1 \r\n\r\nIt is possible to specify somewhere which version of Cdh to install? I don't see an option for that in the Eggo CLI or code. There is no explicit reference to the Cdh version to install (5.4.8)  anywhere.\r\n\r\nMaking it possible to install Cdh 5.5.0 would enable the usage of a lot of features added to spark between 1.3 and and 1.5.\r\n\r\nThank you. \r\n\r\nNeill \r\n\r\n\r\n\r\n \r\n\r\n\r\n\r\n"}}