{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/4487","labels_url":"https://api.github.com/repos/apache/spark/issues/4487/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/4487/comments","events_url":"https://api.github.com/repos/apache/spark/issues/4487/events","html_url":"https://github.com/apache/spark/pull/4487","id":57114442,"number":4487,"title":"[SPARK-5641] Allow spark_ec2.py to copy a wider range of files to cluster via deploy.generic","user":{"login":"florianverhein","id":4240669,"avatar_url":"https://avatars.githubusercontent.com/u/4240669?v=3","gravatar_id":"","url":"https://api.github.com/users/florianverhein","html_url":"https://github.com/florianverhein","followers_url":"https://api.github.com/users/florianverhein/followers","following_url":"https://api.github.com/users/florianverhein/following{/other_user}","gists_url":"https://api.github.com/users/florianverhein/gists{/gist_id}","starred_url":"https://api.github.com/users/florianverhein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/florianverhein/subscriptions","organizations_url":"https://api.github.com/users/florianverhein/orgs","repos_url":"https://api.github.com/users/florianverhein/repos","events_url":"https://api.github.com/users/florianverhein/events{/privacy}","received_events_url":"https://api.github.com/users/florianverhein/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":18,"created_at":"2015-02-10T00:51:45Z","updated_at":"2015-02-12T18:38:14Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/4487","html_url":"https://github.com/apache/spark/pull/4487","diff_url":"https://github.com/apache/spark/pull/4487.diff","patch_url":"https://github.com/apache/spark/pull/4487.patch"},"body":"Currently, deploy_files fails on binary files. This change fixes this for certain file extensions, thus allowing a greater range of files to be deployed to the cluster.  \r\n\r\nThis improvement is really useful if binary files need to be uploaded. E.g. I use this for rpm transfer to install extra stuff at cluster deployment time (I build or grab the rpms just prior and these rpms are not available externally - so this provides a convenient and secure way of delivering extra modules without code changes).\r\n\r\nNote that it could also be used to override what's on the image (e.g. a jar) or to deliver a tar ball (e.g. if it's not on S3 - say, you built spark against a different hadoop or tachyon version)\r\n\r\nTo use this feature, a user can just dump the files into ec2/deploy.generic/ before calling spark_ec2.py.\r\n\r\nDetecting binary files is non-trivial. So instead, the implementation has a list of file extensions that will trigger simple file copying."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/74126044","html_url":"https://github.com/apache/spark/pull/4487#issuecomment-74126044","issue_url":"https://api.github.com/repos/apache/spark/issues/4487","id":74126044,"user":{"login":"shivaram","id":143893,"avatar_url":"https://avatars.githubusercontent.com/u/143893?v=3","gravatar_id":"","url":"https://api.github.com/users/shivaram","html_url":"https://github.com/shivaram","followers_url":"https://api.github.com/users/shivaram/followers","following_url":"https://api.github.com/users/shivaram/following{/other_user}","gists_url":"https://api.github.com/users/shivaram/gists{/gist_id}","starred_url":"https://api.github.com/users/shivaram/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shivaram/subscriptions","organizations_url":"https://api.github.com/users/shivaram/orgs","repos_url":"https://api.github.com/users/shivaram/repos","events_url":"https://api.github.com/users/shivaram/events{/privacy}","received_events_url":"https://api.github.com/users/shivaram/received_events","type":"User","site_admin":false},"created_at":"2015-02-12T18:38:14Z","updated_at":"2015-02-12T18:38:14Z","body":"@florianverhein Sorry for the delay in getting back. At a high level adding a flag to specify a `deploy.generic` directory seems better than asking people to copy binaries. However there are a couple of things that I am still concerned about\r\n\r\n- To do variable substitution we make a copy of files from deploy.generic to a temp directory and then copy files after substitution from the temp directory to the EC2 machine. This might be a bad idea for large rpms / binaries as we will unnecessarily create copies.\r\n\r\n- I am not sure why one would want to add an extra module through deploy.generic. Isn't it easier / better to have a fork of spark-ec2 where the module is placed and then just use `--spark-ec2-git-repo` ? \r\n\r\n- Finally I am not sure I completely get the case for adding new flags (like `TEST_MODULES`). If these flags need to be set based on EC2 instances, then it looks like one does need to edit `spark_ec2.py` -- if not, new flags can be directly added to `ec2-variables.sh` (i.e. by editing the file in the source tree)\r\n"}}