{"action":"created","comment":{"url":"https://api.github.com/repos/linkedin/gobblin/pulls/comments/43663519","id":43663519,"diff_hunk":"@@ -43,7 +47,7 @@\n  */\n public class SftpFsHelper implements SizeAwareFileBasedHelper {","path":"gobblin-core/src/main/java/gobblin/source/extractor/extract/sftp/SftpFsHelper.java","position":37,"original_position":37,"commit_id":"7b972b8c8fda90e9c30fc5727e65e94d9e3548d5","original_commit_id":"7b972b8c8fda90e9c30fc5727e65e94d9e3548d5","user":{"login":"chavdar","id":928795,"avatar_url":"https://avatars.githubusercontent.com/u/928795?v=3","gravatar_id":"","url":"https://api.github.com/users/chavdar","html_url":"https://github.com/chavdar","followers_url":"https://api.github.com/users/chavdar/followers","following_url":"https://api.github.com/users/chavdar/following{/other_user}","gists_url":"https://api.github.com/users/chavdar/gists{/gist_id}","starred_url":"https://api.github.com/users/chavdar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chavdar/subscriptions","organizations_url":"https://api.github.com/users/chavdar/orgs","repos_url":"https://api.github.com/users/chavdar/repos","events_url":"https://api.github.com/users/chavdar/events{/privacy}","received_events_url":"https://api.github.com/users/chavdar/received_events","type":"User","site_admin":false},"body":"I think the name is misleading. Typically, I expect a help class to be a set of static methods. This class does session management so I think it should be named SftpSession.","created_at":"2015-11-02T18:39:52Z","updated_at":"2015-11-02T18:39:52Z","html_url":"https://github.com/linkedin/gobblin/pull/423#discussion_r43663519","pull_request_url":"https://api.github.com/repos/linkedin/gobblin/pulls/423","_links":{"self":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/comments/43663519"},"html":{"href":"https://github.com/linkedin/gobblin/pull/423#discussion_r43663519"},"pull_request":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/423"}}},"pull_request":{"url":"https://api.github.com/repos/linkedin/gobblin/pulls/423","id":49222424,"html_url":"https://github.com/linkedin/gobblin/pull/423","diff_url":"https://github.com/linkedin/gobblin/pull/423.diff","patch_url":"https://github.com/linkedin/gobblin/pull/423.patch","issue_url":"https://api.github.com/repos/linkedin/gobblin/issues/423","number":423,"state":"open","locked":false,"title":"Part2 sftp copy","user":{"login":"pcadabam","id":1166372,"avatar_url":"https://avatars.githubusercontent.com/u/1166372?v=3","gravatar_id":"","url":"https://api.github.com/users/pcadabam","html_url":"https://github.com/pcadabam","followers_url":"https://api.github.com/users/pcadabam/followers","following_url":"https://api.github.com/users/pcadabam/following{/other_user}","gists_url":"https://api.github.com/users/pcadabam/gists{/gist_id}","starred_url":"https://api.github.com/users/pcadabam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcadabam/subscriptions","organizations_url":"https://api.github.com/users/pcadabam/orgs","repos_url":"https://api.github.com/users/pcadabam/repos","events_url":"https://api.github.com/users/pcadabam/events{/privacy}","received_events_url":"https://api.github.com/users/pcadabam/received_events","type":"User","site_admin":false},"body":"This is part 2 of https://github.com/linkedin/gobblin/pull/403\r\n\r\nTasks from meeting, \r\n1. Use password encryption for SFTP creds and decryption passphrase - Done\r\n2. Open ticket to explore unarchival in converter - Done\r\n3. Rename archive..writer to Tar..writer or support other archival types in archive writer -  Done\r\n4. Implement permission -  Done\r\n5. Add preserver flags (group, owner, permission) to job config -  Will be done in config package\r\n6. split pull request to Dataset, Copyable Dataset, SFTP -  Done\r\n\r\nI have added support to handle multiple datasets in the writer and the publisher. The task staging and task output now have the complete path of the final destination under them. \r\nThe publisher goes over each dataset in the task out put and publishes them.\r\n\r\n##### E.g. \r\n\r\nSource FileSystem\r\n/src/a/b/c /t1.txt\r\n/src/a/b/c /t2.txt\r\n\r\nLet's say /src/a/b/c needs to be copied to /dest/data/a/b/c,\r\n\r\n##### After InputStreamDataWriter.write\r\nGobblin's task staging directory looks like,\r\n-/tmp/gobblin/task_staging/task_Id1/dest/data/a/b/c/t1.txt\r\n-/tmp/gobblin/task_staging/task_Id2/dest/data/a/b/c/t2.txt\r\n\r\n##### After InputStreamDataWriter.commit\r\nGobblin's task output directory looks like,\r\n-/tmp/gobblin/task_output/dest/data/a/b/c/t1.txt\r\n-/tmp/gobblin/task_output/dest/data/a/b/c/t2.txt\r\n\r\n##### Note that the commit method accumulates directory paths under multiple task staging directories to one task output directory. This will make sure parallel mapper task DO NOT overwrite directories created by another mapper. \r\n\r\n##### After CopyDataPublisher.publish\r\n\r\nDestination FileSystem looks like,\r\n/dest/data/a/b/c/t1.txt\r\n/dest/data/a/b/c/t2.txt\r\n\r\nEach dataset is renamed at a time by the publisher. In this case /dest/data.\r\n","created_at":"2015-10-30T01:39:54Z","updated_at":"2015-11-02T18:39:52Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c3eea00727047a36b8ef52f17eb658fe62626d55","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/linkedin/gobblin/pulls/423/commits","review_comments_url":"https://api.github.com/repos/linkedin/gobblin/pulls/423/comments","review_comment_url":"https://api.github.com/repos/linkedin/gobblin/pulls/comments{/number}","comments_url":"https://api.github.com/repos/linkedin/gobblin/issues/423/comments","statuses_url":"https://api.github.com/repos/linkedin/gobblin/statuses/7b972b8c8fda90e9c30fc5727e65e94d9e3548d5","head":{"label":"pcadabam:master","ref":"master","sha":"7b972b8c8fda90e9c30fc5727e65e94d9e3548d5","user":{"login":"pcadabam","id":1166372,"avatar_url":"https://avatars.githubusercontent.com/u/1166372?v=3","gravatar_id":"","url":"https://api.github.com/users/pcadabam","html_url":"https://github.com/pcadabam","followers_url":"https://api.github.com/users/pcadabam/followers","following_url":"https://api.github.com/users/pcadabam/following{/other_user}","gists_url":"https://api.github.com/users/pcadabam/gists{/gist_id}","starred_url":"https://api.github.com/users/pcadabam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcadabam/subscriptions","organizations_url":"https://api.github.com/users/pcadabam/orgs","repos_url":"https://api.github.com/users/pcadabam/repos","events_url":"https://api.github.com/users/pcadabam/events{/privacy}","received_events_url":"https://api.github.com/users/pcadabam/received_events","type":"User","site_admin":false},"repo":{"id":36956709,"name":"gobblin","full_name":"pcadabam/gobblin","owner":{"login":"pcadabam","id":1166372,"avatar_url":"https://avatars.githubusercontent.com/u/1166372?v=3","gravatar_id":"","url":"https://api.github.com/users/pcadabam","html_url":"https://github.com/pcadabam","followers_url":"https://api.github.com/users/pcadabam/followers","following_url":"https://api.github.com/users/pcadabam/following{/other_user}","gists_url":"https://api.github.com/users/pcadabam/gists{/gist_id}","starred_url":"https://api.github.com/users/pcadabam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcadabam/subscriptions","organizations_url":"https://api.github.com/users/pcadabam/orgs","repos_url":"https://api.github.com/users/pcadabam/repos","events_url":"https://api.github.com/users/pcadabam/events{/privacy}","received_events_url":"https://api.github.com/users/pcadabam/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/pcadabam/gobblin","description":"","fork":true,"url":"https://api.github.com/repos/pcadabam/gobblin","forks_url":"https://api.github.com/repos/pcadabam/gobblin/forks","keys_url":"https://api.github.com/repos/pcadabam/gobblin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pcadabam/gobblin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pcadabam/gobblin/teams","hooks_url":"https://api.github.com/repos/pcadabam/gobblin/hooks","issue_events_url":"https://api.github.com/repos/pcadabam/gobblin/issues/events{/number}","events_url":"https://api.github.com/repos/pcadabam/gobblin/events","assignees_url":"https://api.github.com/repos/pcadabam/gobblin/assignees{/user}","branches_url":"https://api.github.com/repos/pcadabam/gobblin/branches{/branch}","tags_url":"https://api.github.com/repos/pcadabam/gobblin/tags","blobs_url":"https://api.github.com/repos/pcadabam/gobblin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pcadabam/gobblin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pcadabam/gobblin/git/refs{/sha}","trees_url":"https://api.github.com/repos/pcadabam/gobblin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pcadabam/gobblin/statuses/{sha}","languages_url":"https://api.github.com/repos/pcadabam/gobblin/languages","stargazers_url":"https://api.github.com/repos/pcadabam/gobblin/stargazers","contributors_url":"https://api.github.com/repos/pcadabam/gobblin/contributors","subscribers_url":"https://api.github.com/repos/pcadabam/gobblin/subscribers","subscription_url":"https://api.github.com/repos/pcadabam/gobblin/subscription","commits_url":"https://api.github.com/repos/pcadabam/gobblin/commits{/sha}","git_commits_url":"https://api.github.com/repos/pcadabam/gobblin/git/commits{/sha}","comments_url":"https://api.github.com/repos/pcadabam/gobblin/comments{/number}","issue_comment_url":"https://api.github.com/repos/pcadabam/gobblin/issues/comments{/number}","contents_url":"https://api.github.com/repos/pcadabam/gobblin/contents/{+path}","compare_url":"https://api.github.com/repos/pcadabam/gobblin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pcadabam/gobblin/merges","archive_url":"https://api.github.com/repos/pcadabam/gobblin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pcadabam/gobblin/downloads","issues_url":"https://api.github.com/repos/pcadabam/gobblin/issues{/number}","pulls_url":"https://api.github.com/repos/pcadabam/gobblin/pulls{/number}","milestones_url":"https://api.github.com/repos/pcadabam/gobblin/milestones{/number}","notifications_url":"https://api.github.com/repos/pcadabam/gobblin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pcadabam/gobblin/labels{/name}","releases_url":"https://api.github.com/repos/pcadabam/gobblin/releases{/id}","created_at":"2015-06-05T21:39:40Z","updated_at":"2015-06-05T21:39:42Z","pushed_at":"2015-10-30T01:15:53Z","git_url":"git://github.com/pcadabam/gobblin.git","ssh_url":"git@github.com:pcadabam/gobblin.git","clone_url":"https://github.com/pcadabam/gobblin.git","svn_url":"https://github.com/pcadabam/gobblin","homepage":null,"size":80272,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"linkedin:master","ref":"master","sha":"0b0df2c7bb718d22a6181db258c112229a93e66e","user":{"login":"linkedin","id":357098,"avatar_url":"https://avatars.githubusercontent.com/u/357098?v=3","gravatar_id":"","url":"https://api.github.com/users/linkedin","html_url":"https://github.com/linkedin","followers_url":"https://api.github.com/users/linkedin/followers","following_url":"https://api.github.com/users/linkedin/following{/other_user}","gists_url":"https://api.github.com/users/linkedin/gists{/gist_id}","starred_url":"https://api.github.com/users/linkedin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linkedin/subscriptions","organizations_url":"https://api.github.com/users/linkedin/orgs","repos_url":"https://api.github.com/users/linkedin/repos","events_url":"https://api.github.com/users/linkedin/events{/privacy}","received_events_url":"https://api.github.com/users/linkedin/received_events","type":"Organization","site_admin":false},"repo":{"id":27392543,"name":"gobblin","full_name":"linkedin/gobblin","owner":{"login":"linkedin","id":357098,"avatar_url":"https://avatars.githubusercontent.com/u/357098?v=3","gravatar_id":"","url":"https://api.github.com/users/linkedin","html_url":"https://github.com/linkedin","followers_url":"https://api.github.com/users/linkedin/followers","following_url":"https://api.github.com/users/linkedin/following{/other_user}","gists_url":"https://api.github.com/users/linkedin/gists{/gist_id}","starred_url":"https://api.github.com/users/linkedin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linkedin/subscriptions","organizations_url":"https://api.github.com/users/linkedin/orgs","repos_url":"https://api.github.com/users/linkedin/repos","events_url":"https://api.github.com/users/linkedin/events{/privacy}","received_events_url":"https://api.github.com/users/linkedin/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/linkedin/gobblin","description":"Universal data ingestion framework for Hadoop.","fork":false,"url":"https://api.github.com/repos/linkedin/gobblin","forks_url":"https://api.github.com/repos/linkedin/gobblin/forks","keys_url":"https://api.github.com/repos/linkedin/gobblin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/linkedin/gobblin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/linkedin/gobblin/teams","hooks_url":"https://api.github.com/repos/linkedin/gobblin/hooks","issue_events_url":"https://api.github.com/repos/linkedin/gobblin/issues/events{/number}","events_url":"https://api.github.com/repos/linkedin/gobblin/events","assignees_url":"https://api.github.com/repos/linkedin/gobblin/assignees{/user}","branches_url":"https://api.github.com/repos/linkedin/gobblin/branches{/branch}","tags_url":"https://api.github.com/repos/linkedin/gobblin/tags","blobs_url":"https://api.github.com/repos/linkedin/gobblin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/linkedin/gobblin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/linkedin/gobblin/git/refs{/sha}","trees_url":"https://api.github.com/repos/linkedin/gobblin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/linkedin/gobblin/statuses/{sha}","languages_url":"https://api.github.com/repos/linkedin/gobblin/languages","stargazers_url":"https://api.github.com/repos/linkedin/gobblin/stargazers","contributors_url":"https://api.github.com/repos/linkedin/gobblin/contributors","subscribers_url":"https://api.github.com/repos/linkedin/gobblin/subscribers","subscription_url":"https://api.github.com/repos/linkedin/gobblin/subscription","commits_url":"https://api.github.com/repos/linkedin/gobblin/commits{/sha}","git_commits_url":"https://api.github.com/repos/linkedin/gobblin/git/commits{/sha}","comments_url":"https://api.github.com/repos/linkedin/gobblin/comments{/number}","issue_comment_url":"https://api.github.com/repos/linkedin/gobblin/issues/comments{/number}","contents_url":"https://api.github.com/repos/linkedin/gobblin/contents/{+path}","compare_url":"https://api.github.com/repos/linkedin/gobblin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/linkedin/gobblin/merges","archive_url":"https://api.github.com/repos/linkedin/gobblin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/linkedin/gobblin/downloads","issues_url":"https://api.github.com/repos/linkedin/gobblin/issues{/number}","pulls_url":"https://api.github.com/repos/linkedin/gobblin/pulls{/number}","milestones_url":"https://api.github.com/repos/linkedin/gobblin/milestones{/number}","notifications_url":"https://api.github.com/repos/linkedin/gobblin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/linkedin/gobblin/labels{/name}","releases_url":"https://api.github.com/repos/linkedin/gobblin/releases{/id}","created_at":"2014-12-01T18:10:50Z","updated_at":"2015-10-31T11:25:32Z","pushed_at":"2015-11-02T00:58:24Z","git_url":"git://github.com/linkedin/gobblin.git","ssh_url":"git@github.com:linkedin/gobblin.git","clone_url":"https://github.com/linkedin/gobblin.git","svn_url":"https://github.com/linkedin/gobblin","homepage":"https://github.com/linkedin/gobblin/wiki","size":162074,"stargazers_count":270,"watchers_count":270,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":90,"mirror_url":null,"open_issues_count":15,"forks":90,"open_issues":15,"watchers":270,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/423"},"html":{"href":"https://github.com/linkedin/gobblin/pull/423"},"issue":{"href":"https://api.github.com/repos/linkedin/gobblin/issues/423"},"comments":{"href":"https://api.github.com/repos/linkedin/gobblin/issues/423/comments"},"review_comments":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/423/comments"},"review_comment":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/423/commits"},"statuses":{"href":"https://api.github.com/repos/linkedin/gobblin/statuses/7b972b8c8fda90e9c30fc5727e65e94d9e3548d5"}}}}