{"action":"created","issue":{"url":"https://api.github.com/repos/awslabs/emr-bootstrap-actions/issues/67","labels_url":"https://api.github.com/repos/awslabs/emr-bootstrap-actions/issues/67/labels{/name}","comments_url":"https://api.github.com/repos/awslabs/emr-bootstrap-actions/issues/67/comments","events_url":"https://api.github.com/repos/awslabs/emr-bootstrap-actions/issues/67/events","html_url":"https://github.com/awslabs/emr-bootstrap-actions/issues/67","id":60542868,"number":67,"title":"How is Spark for EMR constructed?  Is there a repo for it and can we build it?","user":{"login":"jkleckner","id":1223485,"avatar_url":"https://avatars.githubusercontent.com/u/1223485?v=3","gravatar_id":"","url":"https://api.github.com/users/jkleckner","html_url":"https://github.com/jkleckner","followers_url":"https://api.github.com/users/jkleckner/followers","following_url":"https://api.github.com/users/jkleckner/following{/other_user}","gists_url":"https://api.github.com/users/jkleckner/gists{/gist_id}","starred_url":"https://api.github.com/users/jkleckner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkleckner/subscriptions","organizations_url":"https://api.github.com/users/jkleckner/orgs","repos_url":"https://api.github.com/users/jkleckner/repos","events_url":"https://api.github.com/users/jkleckner/events{/privacy}","received_events_url":"https://api.github.com/users/jkleckner/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-03-10T18:06:09Z","updated_at":"2015-03-16T18:24:34Z","closed_at":null,"body":"I have a test case that:\r\n\r\n- Test case runs correctly locally when spark is built using sbt\r\n- Test case fails locally when spark is built using mvn\r\n- Test case runs correctly in Amazon versions 1.2.0.a and 1.2.1.a\r\n\r\nObviously the assembly created by maven is different from the one created by sbt.  For example, there is the ability to shade jar files with Maven that isn't possible in sbt and that is used by parquet-column (and perhaps other modules).\r\n\r\nAre the Amazon versions built using maven or sbt?\r\n\r\nDo you make a repo available for how those spark builds are created?\r\n\r\nI would like to see how those are built to figure out what is going on.\r\n\r\nFor double bonus points, I would like to be able to roll my own spark installs to try out in EMR."},"comment":{"url":"https://api.github.com/repos/awslabs/emr-bootstrap-actions/issues/comments/81854212","html_url":"https://github.com/awslabs/emr-bootstrap-actions/issues/67#issuecomment-81854212","issue_url":"https://api.github.com/repos/awslabs/emr-bootstrap-actions/issues/67","id":81854212,"user":{"login":"christopherbozeman","id":55891,"avatar_url":"https://avatars.githubusercontent.com/u/55891?v=3","gravatar_id":"","url":"https://api.github.com/users/christopherbozeman","html_url":"https://github.com/christopherbozeman","followers_url":"https://api.github.com/users/christopherbozeman/followers","following_url":"https://api.github.com/users/christopherbozeman/following{/other_user}","gists_url":"https://api.github.com/users/christopherbozeman/gists{/gist_id}","starred_url":"https://api.github.com/users/christopherbozeman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/christopherbozeman/subscriptions","organizations_url":"https://api.github.com/users/christopherbozeman/orgs","repos_url":"https://api.github.com/users/christopherbozeman/repos","events_url":"https://api.github.com/users/christopherbozeman/events{/privacy}","received_events_url":"https://api.github.com/users/christopherbozeman/received_events","type":"User","site_admin":false},"created_at":"2015-03-16T18:24:34Z","updated_at":"2015-03-16T18:24:34Z","body":"Yes, you can build your own distribution and use a custom `config.file` referenced with the `-c` argument to `install-spark` bootstrap-action for installation."}}