{"action":"created","issue":{"url":"https://api.github.com/repos/bigdatagenomics/eggo/issues/35","labels_url":"https://api.github.com/repos/bigdatagenomics/eggo/issues/35/labels{/name}","comments_url":"https://api.github.com/repos/bigdatagenomics/eggo/issues/35/comments","events_url":"https://api.github.com/repos/bigdatagenomics/eggo/issues/35/events","html_url":"https://github.com/bigdatagenomics/eggo/issues/35","id":69256089,"number":35,"title":"hadoop-streaming dnloader is breaking because the mapper can't get the AWS credentials","user":{"login":"laserson","id":118141,"avatar_url":"https://avatars.githubusercontent.com/u/118141?v=3","gravatar_id":"","url":"https://api.github.com/users/laserson","html_url":"https://github.com/laserson","followers_url":"https://api.github.com/users/laserson/followers","following_url":"https://api.github.com/users/laserson/following{/other_user}","gists_url":"https://api.github.com/users/laserson/gists{/gist_id}","starred_url":"https://api.github.com/users/laserson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laserson/subscriptions","organizations_url":"https://api.github.com/users/laserson/orgs","repos_url":"https://api.github.com/users/laserson/repos","events_url":"https://api.github.com/users/laserson/events{/privacy}","received_events_url":"https://api.github.com/users/laserson/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-04-18T02:19:00Z","updated_at":"2015-04-21T17:05:25Z","closed_at":"2015-04-21T13:20:35Z","body":"We are using hadoop streaming to manage the parallel dnloads of the data sets to S3.  In the process, we are instantiating a Luigi `S3Client` object which needs the AWS credentials.  As written, we try to get these values from the standard environment variables.  However, these environment variables are not available to the streaming mapper.\r\n\r\nOne solution is to set `-cmdenv` vars in the invocation of the streaming job.  However, it appears that Luigi does not support setting these options in `HadoopJobRunner`.  There are probably many workarounds, but the best would be to submit a Luigi patch that lets you set environment vars.\r\n\r\nThoughts, @tomwhite?  Is there a better way to get the AWS credentials to the mapper?"},"comment":{"url":"https://api.github.com/repos/bigdatagenomics/eggo/issues/comments/94872943","html_url":"https://github.com/bigdatagenomics/eggo/issues/35#issuecomment-94872943","issue_url":"https://api.github.com/repos/bigdatagenomics/eggo/issues/35","id":94872943,"user":{"login":"laserson","id":118141,"avatar_url":"https://avatars.githubusercontent.com/u/118141?v=3","gravatar_id":"","url":"https://api.github.com/users/laserson","html_url":"https://github.com/laserson","followers_url":"https://api.github.com/users/laserson/followers","following_url":"https://api.github.com/users/laserson/following{/other_user}","gists_url":"https://api.github.com/users/laserson/gists{/gist_id}","starred_url":"https://api.github.com/users/laserson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laserson/subscriptions","organizations_url":"https://api.github.com/users/laserson/orgs","repos_url":"https://api.github.com/users/laserson/repos","events_url":"https://api.github.com/users/laserson/events{/privacy}","received_events_url":"https://api.github.com/users/laserson/received_events","type":"User","site_admin":false},"created_at":"2015-04-21T17:05:25Z","updated_at":"2015-04-21T17:05:25Z","body":"Awesome! For the record, I think the luigi S3 client either does or can\nwrap the hadoop CLI.\nOn Apr 21, 2015 21:23, \"Tom White\" <notifications@github.com> wrote:\n\n> Thanks for the suggestion @jfeala <https://github.com/jfeala>. I was able\n> to fix this in the end by specifying -cmdenv via the streaming_args\n> parameter in Luigi. I committed this to unblock eggo and get master working\n> again.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/bigdatagenomics/eggo/issues/35#issuecomment-94792627>.\n>\n"}}