{"action":"created","comment":{"url":"https://api.github.com/repos/linkedin/gobblin/pulls/comments/43833065","id":43833065,"diff_hunk":"@@ -0,0 +1,51 @@\n+/*\n+ * Copyright (C) 2014-2015 LinkedIn Corp. All rights reserved.\n+ *\n+ * Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use\n+ * this file except in compliance with the License. You may obtain a copy of the\n+ * License at  http://www.apache.org/licenses/LICENSE-2.0\n+ *\n+ * Unless required by applicable law or agreed to in writing, software distributed\n+ * under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR\n+ * CONDITIONS OF ANY KIND, either express or implied.\n+ */\n+\n+package gobblin.data.management.copy.converter;\n+\n+import gobblin.configuration.WorkUnitState;\n+import gobblin.converter.Converter;\n+import gobblin.converter.DataConversionException;\n+import gobblin.converter.SchemaConversionException;\n+import gobblin.converter.SingleRecordIterable;\n+import gobblin.data.management.copy.FileAwareInputStream;\n+\n+import java.io.IOException;\n+import java.io.InputStream;\n+import java.util.zip.GZIPInputStream;\n+\n+\n+/**\n+ * A {@link Converter} that converts an archived {@link InputStream} to a tar {@link InputStream}. Wraps the given\n+ * archived {@link InputStream} with {@link GZIPInputStream} Use this converter if the {@link InputStream} from source\n+ * is compressed.\n+ */\n+public class UnArchiveConverter extends Converter<String, String, FileAwareInputStream, FileAwareInputStream> {","path":"gobblin-data-management/src/main/java/gobblin/data/management/copy/converter/UnArchiveConverter.java","position":32,"original_position":32,"commit_id":"7b972b8c8fda90e9c30fc5727e65e94d9e3548d5","original_commit_id":"7b972b8c8fda90e9c30fc5727e65e94d9e3548d5","user":{"login":"pcadabam","id":1166372,"avatar_url":"https://avatars.githubusercontent.com/u/1166372?v=3","gravatar_id":"","url":"https://api.github.com/users/pcadabam","html_url":"https://github.com/pcadabam","followers_url":"https://api.github.com/users/pcadabam/followers","following_url":"https://api.github.com/users/pcadabam/following{/other_user}","gists_url":"https://api.github.com/users/pcadabam/gists{/gist_id}","starred_url":"https://api.github.com/users/pcadabam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcadabam/subscriptions","organizations_url":"https://api.github.com/users/pcadabam/orgs","repos_url":"https://api.github.com/users/pcadabam/repos","events_url":"https://api.github.com/users/pcadabam/events{/privacy}","received_events_url":"https://api.github.com/users/pcadabam/received_events","type":"User","site_admin":false},"body":"I have made the UnArchiveConverter to support both gzip and zip streams. I think we can avoid another config key. I have implemented an UnzipConverter and UnGzipConverter. If the user wants to specify the inflater alogorithm he can do so by specifying the right converter.\r\nThe UnArchiveConverter will look at the extension and call UnZipConverter/UnGzipConverter by extension.","created_at":"2015-11-04T01:39:28Z","updated_at":"2015-11-04T01:39:28Z","html_url":"https://github.com/linkedin/gobblin/pull/423#discussion_r43833065","pull_request_url":"https://api.github.com/repos/linkedin/gobblin/pulls/423","_links":{"self":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/comments/43833065"},"html":{"href":"https://github.com/linkedin/gobblin/pull/423#discussion_r43833065"},"pull_request":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/423"}}},"pull_request":{"url":"https://api.github.com/repos/linkedin/gobblin/pulls/423","id":49222424,"html_url":"https://github.com/linkedin/gobblin/pull/423","diff_url":"https://github.com/linkedin/gobblin/pull/423.diff","patch_url":"https://github.com/linkedin/gobblin/pull/423.patch","issue_url":"https://api.github.com/repos/linkedin/gobblin/issues/423","number":423,"state":"open","locked":false,"title":"Part2 sftp copy","user":{"login":"pcadabam","id":1166372,"avatar_url":"https://avatars.githubusercontent.com/u/1166372?v=3","gravatar_id":"","url":"https://api.github.com/users/pcadabam","html_url":"https://github.com/pcadabam","followers_url":"https://api.github.com/users/pcadabam/followers","following_url":"https://api.github.com/users/pcadabam/following{/other_user}","gists_url":"https://api.github.com/users/pcadabam/gists{/gist_id}","starred_url":"https://api.github.com/users/pcadabam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcadabam/subscriptions","organizations_url":"https://api.github.com/users/pcadabam/orgs","repos_url":"https://api.github.com/users/pcadabam/repos","events_url":"https://api.github.com/users/pcadabam/events{/privacy}","received_events_url":"https://api.github.com/users/pcadabam/received_events","type":"User","site_admin":false},"body":"This is part 2 of https://github.com/linkedin/gobblin/pull/403\r\n\r\nTasks from meeting, \r\n1. Use password encryption for SFTP creds and decryption passphrase - Done\r\n2. Open ticket to explore unarchival in converter - Done\r\n3. Rename archive..writer to Tar..writer or support other archival types in archive writer -  Done\r\n4. Implement permission -  Done\r\n5. Add preserver flags (group, owner, permission) to job config -  Will be done in config package\r\n6. split pull request to Dataset, Copyable Dataset, SFTP -  Done\r\n\r\nI have added support to handle multiple datasets in the writer and the publisher. The task staging and task output now have the complete path of the final destination under them. \r\nThe publisher goes over each dataset in the task out put and publishes them.\r\n\r\n##### E.g. \r\n\r\nSource FileSystem\r\n/src/a/b/c /t1.txt\r\n/src/a/b/c /t2.txt\r\n\r\nLet's say /src/a/b/c needs to be copied to /dest/data/a/b/c,\r\n\r\n##### After InputStreamDataWriter.write\r\nGobblin's task staging directory looks like,\r\n-/tmp/gobblin/task_staging/task_Id1/dest/data/a/b/c/t1.txt\r\n-/tmp/gobblin/task_staging/task_Id2/dest/data/a/b/c/t2.txt\r\n\r\n##### After InputStreamDataWriter.commit\r\nGobblin's task output directory looks like,\r\n-/tmp/gobblin/task_output/dest/data/a/b/c/t1.txt\r\n-/tmp/gobblin/task_output/dest/data/a/b/c/t2.txt\r\n\r\n##### Note that the commit method accumulates directory paths under multiple task staging directories to one task output directory. This will make sure parallel mapper task DO NOT overwrite directories created by another mapper. \r\n\r\n##### After CopyDataPublisher.publish\r\n\r\nDestination FileSystem looks like,\r\n/dest/data/a/b/c/t1.txt\r\n/dest/data/a/b/c/t2.txt\r\n\r\nEach dataset is renamed at a time by the publisher. In this case /dest/data.\r\n","created_at":"2015-10-30T01:39:54Z","updated_at":"2015-11-04T01:39:28Z","closed_at":null,"merged_at":null,"merge_commit_sha":"cfc2a596cf45f067a161fc28988de03d66149f46","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/linkedin/gobblin/pulls/423/commits","review_comments_url":"https://api.github.com/repos/linkedin/gobblin/pulls/423/comments","review_comment_url":"https://api.github.com/repos/linkedin/gobblin/pulls/comments{/number}","comments_url":"https://api.github.com/repos/linkedin/gobblin/issues/423/comments","statuses_url":"https://api.github.com/repos/linkedin/gobblin/statuses/7b972b8c8fda90e9c30fc5727e65e94d9e3548d5","head":{"label":"pcadabam:master","ref":"master","sha":"7b972b8c8fda90e9c30fc5727e65e94d9e3548d5","user":{"login":"pcadabam","id":1166372,"avatar_url":"https://avatars.githubusercontent.com/u/1166372?v=3","gravatar_id":"","url":"https://api.github.com/users/pcadabam","html_url":"https://github.com/pcadabam","followers_url":"https://api.github.com/users/pcadabam/followers","following_url":"https://api.github.com/users/pcadabam/following{/other_user}","gists_url":"https://api.github.com/users/pcadabam/gists{/gist_id}","starred_url":"https://api.github.com/users/pcadabam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcadabam/subscriptions","organizations_url":"https://api.github.com/users/pcadabam/orgs","repos_url":"https://api.github.com/users/pcadabam/repos","events_url":"https://api.github.com/users/pcadabam/events{/privacy}","received_events_url":"https://api.github.com/users/pcadabam/received_events","type":"User","site_admin":false},"repo":{"id":36956709,"name":"gobblin","full_name":"pcadabam/gobblin","owner":{"login":"pcadabam","id":1166372,"avatar_url":"https://avatars.githubusercontent.com/u/1166372?v=3","gravatar_id":"","url":"https://api.github.com/users/pcadabam","html_url":"https://github.com/pcadabam","followers_url":"https://api.github.com/users/pcadabam/followers","following_url":"https://api.github.com/users/pcadabam/following{/other_user}","gists_url":"https://api.github.com/users/pcadabam/gists{/gist_id}","starred_url":"https://api.github.com/users/pcadabam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcadabam/subscriptions","organizations_url":"https://api.github.com/users/pcadabam/orgs","repos_url":"https://api.github.com/users/pcadabam/repos","events_url":"https://api.github.com/users/pcadabam/events{/privacy}","received_events_url":"https://api.github.com/users/pcadabam/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/pcadabam/gobblin","description":"","fork":true,"url":"https://api.github.com/repos/pcadabam/gobblin","forks_url":"https://api.github.com/repos/pcadabam/gobblin/forks","keys_url":"https://api.github.com/repos/pcadabam/gobblin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pcadabam/gobblin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pcadabam/gobblin/teams","hooks_url":"https://api.github.com/repos/pcadabam/gobblin/hooks","issue_events_url":"https://api.github.com/repos/pcadabam/gobblin/issues/events{/number}","events_url":"https://api.github.com/repos/pcadabam/gobblin/events","assignees_url":"https://api.github.com/repos/pcadabam/gobblin/assignees{/user}","branches_url":"https://api.github.com/repos/pcadabam/gobblin/branches{/branch}","tags_url":"https://api.github.com/repos/pcadabam/gobblin/tags","blobs_url":"https://api.github.com/repos/pcadabam/gobblin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pcadabam/gobblin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pcadabam/gobblin/git/refs{/sha}","trees_url":"https://api.github.com/repos/pcadabam/gobblin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pcadabam/gobblin/statuses/{sha}","languages_url":"https://api.github.com/repos/pcadabam/gobblin/languages","stargazers_url":"https://api.github.com/repos/pcadabam/gobblin/stargazers","contributors_url":"https://api.github.com/repos/pcadabam/gobblin/contributors","subscribers_url":"https://api.github.com/repos/pcadabam/gobblin/subscribers","subscription_url":"https://api.github.com/repos/pcadabam/gobblin/subscription","commits_url":"https://api.github.com/repos/pcadabam/gobblin/commits{/sha}","git_commits_url":"https://api.github.com/repos/pcadabam/gobblin/git/commits{/sha}","comments_url":"https://api.github.com/repos/pcadabam/gobblin/comments{/number}","issue_comment_url":"https://api.github.com/repos/pcadabam/gobblin/issues/comments{/number}","contents_url":"https://api.github.com/repos/pcadabam/gobblin/contents/{+path}","compare_url":"https://api.github.com/repos/pcadabam/gobblin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pcadabam/gobblin/merges","archive_url":"https://api.github.com/repos/pcadabam/gobblin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pcadabam/gobblin/downloads","issues_url":"https://api.github.com/repos/pcadabam/gobblin/issues{/number}","pulls_url":"https://api.github.com/repos/pcadabam/gobblin/pulls{/number}","milestones_url":"https://api.github.com/repos/pcadabam/gobblin/milestones{/number}","notifications_url":"https://api.github.com/repos/pcadabam/gobblin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pcadabam/gobblin/labels{/name}","releases_url":"https://api.github.com/repos/pcadabam/gobblin/releases{/id}","created_at":"2015-06-05T21:39:40Z","updated_at":"2015-06-05T21:39:42Z","pushed_at":"2015-10-30T01:15:53Z","git_url":"git://github.com/pcadabam/gobblin.git","ssh_url":"git@github.com:pcadabam/gobblin.git","clone_url":"https://github.com/pcadabam/gobblin.git","svn_url":"https://github.com/pcadabam/gobblin","homepage":null,"size":80342,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"linkedin:master","ref":"master","sha":"0b0df2c7bb718d22a6181db258c112229a93e66e","user":{"login":"linkedin","id":357098,"avatar_url":"https://avatars.githubusercontent.com/u/357098?v=3","gravatar_id":"","url":"https://api.github.com/users/linkedin","html_url":"https://github.com/linkedin","followers_url":"https://api.github.com/users/linkedin/followers","following_url":"https://api.github.com/users/linkedin/following{/other_user}","gists_url":"https://api.github.com/users/linkedin/gists{/gist_id}","starred_url":"https://api.github.com/users/linkedin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linkedin/subscriptions","organizations_url":"https://api.github.com/users/linkedin/orgs","repos_url":"https://api.github.com/users/linkedin/repos","events_url":"https://api.github.com/users/linkedin/events{/privacy}","received_events_url":"https://api.github.com/users/linkedin/received_events","type":"Organization","site_admin":false},"repo":{"id":27392543,"name":"gobblin","full_name":"linkedin/gobblin","owner":{"login":"linkedin","id":357098,"avatar_url":"https://avatars.githubusercontent.com/u/357098?v=3","gravatar_id":"","url":"https://api.github.com/users/linkedin","html_url":"https://github.com/linkedin","followers_url":"https://api.github.com/users/linkedin/followers","following_url":"https://api.github.com/users/linkedin/following{/other_user}","gists_url":"https://api.github.com/users/linkedin/gists{/gist_id}","starred_url":"https://api.github.com/users/linkedin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linkedin/subscriptions","organizations_url":"https://api.github.com/users/linkedin/orgs","repos_url":"https://api.github.com/users/linkedin/repos","events_url":"https://api.github.com/users/linkedin/events{/privacy}","received_events_url":"https://api.github.com/users/linkedin/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/linkedin/gobblin","description":"Universal data ingestion framework for Hadoop.","fork":false,"url":"https://api.github.com/repos/linkedin/gobblin","forks_url":"https://api.github.com/repos/linkedin/gobblin/forks","keys_url":"https://api.github.com/repos/linkedin/gobblin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/linkedin/gobblin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/linkedin/gobblin/teams","hooks_url":"https://api.github.com/repos/linkedin/gobblin/hooks","issue_events_url":"https://api.github.com/repos/linkedin/gobblin/issues/events{/number}","events_url":"https://api.github.com/repos/linkedin/gobblin/events","assignees_url":"https://api.github.com/repos/linkedin/gobblin/assignees{/user}","branches_url":"https://api.github.com/repos/linkedin/gobblin/branches{/branch}","tags_url":"https://api.github.com/repos/linkedin/gobblin/tags","blobs_url":"https://api.github.com/repos/linkedin/gobblin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/linkedin/gobblin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/linkedin/gobblin/git/refs{/sha}","trees_url":"https://api.github.com/repos/linkedin/gobblin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/linkedin/gobblin/statuses/{sha}","languages_url":"https://api.github.com/repos/linkedin/gobblin/languages","stargazers_url":"https://api.github.com/repos/linkedin/gobblin/stargazers","contributors_url":"https://api.github.com/repos/linkedin/gobblin/contributors","subscribers_url":"https://api.github.com/repos/linkedin/gobblin/subscribers","subscription_url":"https://api.github.com/repos/linkedin/gobblin/subscription","commits_url":"https://api.github.com/repos/linkedin/gobblin/commits{/sha}","git_commits_url":"https://api.github.com/repos/linkedin/gobblin/git/commits{/sha}","comments_url":"https://api.github.com/repos/linkedin/gobblin/comments{/number}","issue_comment_url":"https://api.github.com/repos/linkedin/gobblin/issues/comments{/number}","contents_url":"https://api.github.com/repos/linkedin/gobblin/contents/{+path}","compare_url":"https://api.github.com/repos/linkedin/gobblin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/linkedin/gobblin/merges","archive_url":"https://api.github.com/repos/linkedin/gobblin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/linkedin/gobblin/downloads","issues_url":"https://api.github.com/repos/linkedin/gobblin/issues{/number}","pulls_url":"https://api.github.com/repos/linkedin/gobblin/pulls{/number}","milestones_url":"https://api.github.com/repos/linkedin/gobblin/milestones{/number}","notifications_url":"https://api.github.com/repos/linkedin/gobblin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/linkedin/gobblin/labels{/name}","releases_url":"https://api.github.com/repos/linkedin/gobblin/releases{/id}","created_at":"2014-12-01T18:10:50Z","updated_at":"2015-11-03T09:25:23Z","pushed_at":"2015-11-04T01:00:07Z","git_url":"git://github.com/linkedin/gobblin.git","ssh_url":"git@github.com:linkedin/gobblin.git","clone_url":"https://github.com/linkedin/gobblin.git","svn_url":"https://github.com/linkedin/gobblin","homepage":"https://github.com/linkedin/gobblin/wiki","size":162411,"stargazers_count":271,"watchers_count":271,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":92,"mirror_url":null,"open_issues_count":14,"forks":92,"open_issues":14,"watchers":271,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/423"},"html":{"href":"https://github.com/linkedin/gobblin/pull/423"},"issue":{"href":"https://api.github.com/repos/linkedin/gobblin/issues/423"},"comments":{"href":"https://api.github.com/repos/linkedin/gobblin/issues/423/comments"},"review_comments":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/423/comments"},"review_comment":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/423/commits"},"statuses":{"href":"https://api.github.com/repos/linkedin/gobblin/statuses/7b972b8c8fda90e9c30fc5727e65e94d9e3548d5"}}}}