{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/3670","labels_url":"https://api.github.com/repos/apache/spark/issues/3670/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/3670/comments","events_url":"https://api.github.com/repos/apache/spark/issues/3670/events","html_url":"https://github.com/apache/spark/pull/3670","id":51632071,"number":3670,"title":"SPARK-4687. [WIP] Add an addDirectory API","user":{"login":"sryza","id":654855,"avatar_url":"https://avatars.githubusercontent.com/u/654855?v=3","gravatar_id":"","url":"https://api.github.com/users/sryza","html_url":"https://github.com/sryza","followers_url":"https://api.github.com/users/sryza/followers","following_url":"https://api.github.com/users/sryza/following{/other_user}","gists_url":"https://api.github.com/users/sryza/gists{/gist_id}","starred_url":"https://api.github.com/users/sryza/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sryza/subscriptions","organizations_url":"https://api.github.com/users/sryza/orgs","repos_url":"https://api.github.com/users/sryza/repos","events_url":"https://api.github.com/users/sryza/events{/privacy}","received_events_url":"https://api.github.com/users/sryza/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":20,"created_at":"2014-12-11T00:50:27Z","updated_at":"2015-01-16T10:31:45Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/3670","html_url":"https://github.com/apache/spark/pull/3670","diff_url":"https://github.com/apache/spark/pull/3670.diff","patch_url":"https://github.com/apache/spark/pull/3670.patch"},"body":"This adds an addDirectory API to satisfy Hive's needs.  It only allows specifying HDFS dirs that will be copied down on every executor.\r\n\r\nThere are a couple outstanding questions.\r\n* Should we allow specifying local dirs as well?  The best way to do this would probably be to archive them.  The drawback is that it would require a fair bit of code that I don't know of any current use cases for.\r\n* The addFiles implementation has a caching component that I don't entirely understand.  What events are we caching between?  AFAICT it's users calling addFile on the same file in the same app at different times?  Do we want/need to add something similar for addDirectory.\r\n*  The addFiles implementation will check to see if an added file already exists and has the same contents.  I imagine we want the same behavior, so planning to add this unless people think otherwise.\r\n\r\nI plan to add some tests if people are OK with the approach."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/70235440","html_url":"https://github.com/apache/spark/pull/3670#issuecomment-70235440","issue_url":"https://api.github.com/repos/apache/spark/issues/3670","id":70235440,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2015-01-16T10:31:45Z","updated_at":"2015-01-16T10:31:45Z","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25650/\nTest FAILed."}}