{"action":"created","comment":{"url":"https://api.github.com/repos/polydawn/repeatr/pulls/comments/30013883","id":30013883,"diff_hunk":"@@ -0,0 +1,190 @@\n+package s3\n+\n+import (\n+\t\"bytes\"\n+\t\"crypto/sha512\"\n+\t\"encoding/xml\"\n+\t\"fmt\"\n+\t\"hash\"\n+\t\"io\"\n+\t\"io/ioutil\"\n+\t\"net/http\"\n+\t\"net/url\"\n+\t\"path\"\n+\t\"time\"\n+\n+\t\"github.com/rlmcpherson/s3gof3r\"\n+\t\"github.com/spacemonkeygo/errors\"\n+\t\"github.com/spacemonkeygo/errors/try\"\n+\t\"polydawn.net/repeatr/def\"\n+\t\"polydawn.net/repeatr/lib/guid\"\n+\t\"polydawn.net/repeatr/output\"\n+\t\"polydawn.net/repeatr/output/tar2\"\n+)\n+\n+const Type = \"s3\"\n+\n+var _ output.Output = &Output{} // interface assertion\n+\n+/*\n+\tAmazon S3 silos are used by this transport in a very MVP-oriented way:\n+\ttarballs streamed in and out, because we know how to preserve file\n+\tattributes in a widely-understood way by doing this.\n+\n+\tThis IO system happens to share the same hash-space as the Tar IO system,\n+\tand may thus safely share a cache with Tar IO systems.\n+\n+\tWorking with S3 permissions: good luck; be advised that in many configurations,\n+\taccounts may be able to write data which they then instantly lose permission to\n+\tmanage or even read back; this will result in permission denied errors from this\n+\tsystem, and may result in dirty state left behind (because we have to permission\n+\tto clean it up once we've created this, and cannot discover this until it's\n+\ttoo late!).  PRs for improving this behavior deeply welcome; this author\n+\tis not an S3/AWS/IAM professional.\n+\n+\tTry the docs here: https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html --\n+\tthese describe attaching policies to a bucket that can grant permissions to a user account.","path":"output/s3/s3_output.go","position":46,"original_position":46,"commit_id":"b0a8d01e9708f2c90130ad8163ac75247ac14e96","original_commit_id":"b0a8d01e9708f2c90130ad8163ac75247ac14e96","user":{"login":"kofalt","id":265071,"avatar_url":"https://avatars.githubusercontent.com/u/265071?v=3","gravatar_id":"","url":"https://api.github.com/users/kofalt","html_url":"https://github.com/kofalt","followers_url":"https://api.github.com/users/kofalt/followers","following_url":"https://api.github.com/users/kofalt/following{/other_user}","gists_url":"https://api.github.com/users/kofalt/gists{/gist_id}","starred_url":"https://api.github.com/users/kofalt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kofalt/subscriptions","organizations_url":"https://api.github.com/users/kofalt/orgs","repos_url":"https://api.github.com/users/kofalt/repos","events_url":"https://api.github.com/users/kofalt/events{/privacy}","received_events_url":"https://api.github.com/users/kofalt/received_events","type":"User","site_admin":false},"body":"This is excellent, and I'd like to see either of these added:\r\n\r\n1. Parts of the [PR submission comment](https://github.com/polydawn/repeatr/pull/42#issue-72183745) copied into a comment on the s3 input struct, or\r\n1. Just add a link to said comment, until we get our act together documentation-wide\r\n\r\nThus hopefully alleviating \"how does this work why no docs.\"","created_at":"2015-05-11T05:35:58Z","updated_at":"2015-05-11T05:35:58Z","html_url":"https://github.com/polydawn/repeatr/pull/42#discussion_r30013883","pull_request_url":"https://api.github.com/repos/polydawn/repeatr/pulls/42","_links":{"self":{"href":"https://api.github.com/repos/polydawn/repeatr/pulls/comments/30013883"},"html":{"href":"https://github.com/polydawn/repeatr/pull/42#discussion_r30013883"},"pull_request":{"href":"https://api.github.com/repos/polydawn/repeatr/pulls/42"}}},"pull_request":{"url":"https://api.github.com/repos/polydawn/repeatr/pulls/42","id":34466327,"html_url":"https://github.com/polydawn/repeatr/pull/42","diff_url":"https://github.com/polydawn/repeatr/pull/42.diff","patch_url":"https://github.com/polydawn/repeatr/pull/42.patch","issue_url":"https://api.github.com/repos/polydawn/repeatr/issues/42","number":42,"state":"open","locked":false,"title":"S3 IO","user":{"login":"heavenlyhash","id":627638,"avatar_url":"https://avatars.githubusercontent.com/u/627638?v=3","gravatar_id":"","url":"https://api.github.com/users/heavenlyhash","html_url":"https://github.com/heavenlyhash","followers_url":"https://api.github.com/users/heavenlyhash/followers","following_url":"https://api.github.com/users/heavenlyhash/following{/other_user}","gists_url":"https://api.github.com/users/heavenlyhash/gists{/gist_id}","starred_url":"https://api.github.com/users/heavenlyhash/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/heavenlyhash/subscriptions","organizations_url":"https://api.github.com/users/heavenlyhash/orgs","repos_url":"https://api.github.com/users/heavenlyhash/repos","events_url":"https://api.github.com/users/heavenlyhash/events{/privacy}","received_events_url":"https://api.github.com/users/heavenlyhash/received_events","type":"User","site_admin":false},"body":"MVP on using s3 as a data silo.\r\n\r\nS3 is a k/v store and not an immediate mapping to a filesystem (despite what the UI may seem to suggest) -- to map to a full filesystem we need more bits for file attributes, etc.  This implementation uses tars as the metaphor for mapping filesystems onto s3.  (Another option would be using something like https://github.com/s3fs-fuse/s3fs-fuse -- this project has its own metaphor for mapping filesystems onto s3.  We decided not to use that for MVP because it's a much more complicated system; with this tarball approach, we know we can localize all network work before and after the job, rather than have interesting issues with random latency on disk mid-job.  http://dsl-wiki.cs.uchicago.edu/index.php/Performance_Comparison:Remote_Usage%2C_NFS%2C_S3-fuse%2C_EBS has some other interesting performance notes; there's no short answers to what's \"fastest\", apparently.  We can make more/different implementations in the future.)\r\n\r\nhttps://github.com/rlmcpherson/s3gof3r was used to do the data shoveling and aws attribute auth signing stuff; appears to work really well.\r\n\r\nConfiguration of AWS secret tokens and keys is handled through environment variables for the time being.  We will probably want to expand on this later, building a general secrets management system, for at least two reasons: env vars placed before the entire repeatr system do not have the greatest scope limitations, and also they're effectively a global (what if someone wants to use two different s3 accounts?).  Works for now.\r\n\r\nIncludes a bunch of enhancements to the flexibility of the IO system shared test specs.  Tests now accept parameters for URIs.\r\n\r\nThis also demos the use of different URI schemes to switch storage behavior.  The \"s3://\" scheme will cause the rest of the URI location to be taken literally; the \"s3+splay://\" scheme will instead enable content-addressable naming of uploads, which means the same location string can be used for as a prefix for as much data as you want, and it will DTRT.  (Presuming this goes well, we should consider using URI schemes in the other transports too; this would also let us drop the current \"type\" field assertions and have transports check they recognize URI schemes, which makes a lot more sense when we get to configurable/pluginable dispatches for the IO transport systems.)  We've also talked about upgrading data location URIs to more complex structural data than strings can handle, but without any use cases or specific demos on the table yet, well... URIs still work well enough for jdbc after a decade or so, and we're not so different.\r\n\r\nSetting up S3/AWS permissions models are *not* handled in this.  Those are arcana that should be managed by your organization, not repeatr.","created_at":"2015-04-30T14:43:20Z","updated_at":"2015-05-11T05:35:58Z","closed_at":null,"merged_at":null,"merge_commit_sha":"cec3dbc25661312a56a73d12917771ef1a1799b9","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/polydawn/repeatr/pulls/42/commits","review_comments_url":"https://api.github.com/repos/polydawn/repeatr/pulls/42/comments","review_comment_url":"https://api.github.com/repos/polydawn/repeatr/pulls/comments{/number}","comments_url":"https://api.github.com/repos/polydawn/repeatr/issues/42/comments","statuses_url":"https://api.github.com/repos/polydawn/repeatr/statuses/b0a8d01e9708f2c90130ad8163ac75247ac14e96","head":{"label":"polydawn:s3","ref":"s3","sha":"b0a8d01e9708f2c90130ad8163ac75247ac14e96","user":{"login":"polydawn","id":5307883,"avatar_url":"https://avatars.githubusercontent.com/u/5307883?v=3","gravatar_id":"","url":"https://api.github.com/users/polydawn","html_url":"https://github.com/polydawn","followers_url":"https://api.github.com/users/polydawn/followers","following_url":"https://api.github.com/users/polydawn/following{/other_user}","gists_url":"https://api.github.com/users/polydawn/gists{/gist_id}","starred_url":"https://api.github.com/users/polydawn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/polydawn/subscriptions","organizations_url":"https://api.github.com/users/polydawn/orgs","repos_url":"https://api.github.com/users/polydawn/repos","events_url":"https://api.github.com/users/polydawn/events{/privacy}","received_events_url":"https://api.github.com/users/polydawn/received_events","type":"Organization","site_admin":false},"repo":{"id":31432509,"name":"repeatr","full_name":"polydawn/repeatr","owner":{"login":"polydawn","id":5307883,"avatar_url":"https://avatars.githubusercontent.com/u/5307883?v=3","gravatar_id":"","url":"https://api.github.com/users/polydawn","html_url":"https://github.com/polydawn","followers_url":"https://api.github.com/users/polydawn/followers","following_url":"https://api.github.com/users/polydawn/following{/other_user}","gists_url":"https://api.github.com/users/polydawn/gists{/gist_id}","starred_url":"https://api.github.com/users/polydawn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/polydawn/subscriptions","organizations_url":"https://api.github.com/users/polydawn/orgs","repos_url":"https://api.github.com/users/polydawn/repos","events_url":"https://api.github.com/users/polydawn/events{/privacy}","received_events_url":"https://api.github.com/users/polydawn/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/polydawn/repeatr","description":"Run it. Run it again.","fork":false,"url":"https://api.github.com/repos/polydawn/repeatr","forks_url":"https://api.github.com/repos/polydawn/repeatr/forks","keys_url":"https://api.github.com/repos/polydawn/repeatr/keys{/key_id}","collaborators_url":"https://api.github.com/repos/polydawn/repeatr/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/polydawn/repeatr/teams","hooks_url":"https://api.github.com/repos/polydawn/repeatr/hooks","issue_events_url":"https://api.github.com/repos/polydawn/repeatr/issues/events{/number}","events_url":"https://api.github.com/repos/polydawn/repeatr/events","assignees_url":"https://api.github.com/repos/polydawn/repeatr/assignees{/user}","branches_url":"https://api.github.com/repos/polydawn/repeatr/branches{/branch}","tags_url":"https://api.github.com/repos/polydawn/repeatr/tags","blobs_url":"https://api.github.com/repos/polydawn/repeatr/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/polydawn/repeatr/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/polydawn/repeatr/git/refs{/sha}","trees_url":"https://api.github.com/repos/polydawn/repeatr/git/trees{/sha}","statuses_url":"https://api.github.com/repos/polydawn/repeatr/statuses/{sha}","languages_url":"https://api.github.com/repos/polydawn/repeatr/languages","stargazers_url":"https://api.github.com/repos/polydawn/repeatr/stargazers","contributors_url":"https://api.github.com/repos/polydawn/repeatr/contributors","subscribers_url":"https://api.github.com/repos/polydawn/repeatr/subscribers","subscription_url":"https://api.github.com/repos/polydawn/repeatr/subscription","commits_url":"https://api.github.com/repos/polydawn/repeatr/commits{/sha}","git_commits_url":"https://api.github.com/repos/polydawn/repeatr/git/commits{/sha}","comments_url":"https://api.github.com/repos/polydawn/repeatr/comments{/number}","issue_comment_url":"https://api.github.com/repos/polydawn/repeatr/issues/comments{/number}","contents_url":"https://api.github.com/repos/polydawn/repeatr/contents/{+path}","compare_url":"https://api.github.com/repos/polydawn/repeatr/compare/{base}...{head}","merges_url":"https://api.github.com/repos/polydawn/repeatr/merges","archive_url":"https://api.github.com/repos/polydawn/repeatr/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/polydawn/repeatr/downloads","issues_url":"https://api.github.com/repos/polydawn/repeatr/issues{/number}","pulls_url":"https://api.github.com/repos/polydawn/repeatr/pulls{/number}","milestones_url":"https://api.github.com/repos/polydawn/repeatr/milestones{/number}","notifications_url":"https://api.github.com/repos/polydawn/repeatr/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/polydawn/repeatr/labels{/name}","releases_url":"https://api.github.com/repos/polydawn/repeatr/releases{/id}","created_at":"2015-02-27T18:12:08Z","updated_at":"2015-05-05T03:05:10Z","pushed_at":"2015-05-05T04:09:53Z","git_url":"git://github.com/polydawn/repeatr.git","ssh_url":"git@github.com:polydawn/repeatr.git","clone_url":"https://github.com/polydawn/repeatr.git","svn_url":"https://github.com/polydawn/repeatr","homepage":null,"size":2225,"stargazers_count":3,"watchers_count":3,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":3,"forks":1,"open_issues":3,"watchers":3,"default_branch":"master"}},"base":{"label":"polydawn:master","ref":"master","sha":"baa3e0bc60359afb334d17bfe1e88e389f1e2507","user":{"login":"polydawn","id":5307883,"avatar_url":"https://avatars.githubusercontent.com/u/5307883?v=3","gravatar_id":"","url":"https://api.github.com/users/polydawn","html_url":"https://github.com/polydawn","followers_url":"https://api.github.com/users/polydawn/followers","following_url":"https://api.github.com/users/polydawn/following{/other_user}","gists_url":"https://api.github.com/users/polydawn/gists{/gist_id}","starred_url":"https://api.github.com/users/polydawn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/polydawn/subscriptions","organizations_url":"https://api.github.com/users/polydawn/orgs","repos_url":"https://api.github.com/users/polydawn/repos","events_url":"https://api.github.com/users/polydawn/events{/privacy}","received_events_url":"https://api.github.com/users/polydawn/received_events","type":"Organization","site_admin":false},"repo":{"id":31432509,"name":"repeatr","full_name":"polydawn/repeatr","owner":{"login":"polydawn","id":5307883,"avatar_url":"https://avatars.githubusercontent.com/u/5307883?v=3","gravatar_id":"","url":"https://api.github.com/users/polydawn","html_url":"https://github.com/polydawn","followers_url":"https://api.github.com/users/polydawn/followers","following_url":"https://api.github.com/users/polydawn/following{/other_user}","gists_url":"https://api.github.com/users/polydawn/gists{/gist_id}","starred_url":"https://api.github.com/users/polydawn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/polydawn/subscriptions","organizations_url":"https://api.github.com/users/polydawn/orgs","repos_url":"https://api.github.com/users/polydawn/repos","events_url":"https://api.github.com/users/polydawn/events{/privacy}","received_events_url":"https://api.github.com/users/polydawn/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/polydawn/repeatr","description":"Run it. Run it again.","fork":false,"url":"https://api.github.com/repos/polydawn/repeatr","forks_url":"https://api.github.com/repos/polydawn/repeatr/forks","keys_url":"https://api.github.com/repos/polydawn/repeatr/keys{/key_id}","collaborators_url":"https://api.github.com/repos/polydawn/repeatr/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/polydawn/repeatr/teams","hooks_url":"https://api.github.com/repos/polydawn/repeatr/hooks","issue_events_url":"https://api.github.com/repos/polydawn/repeatr/issues/events{/number}","events_url":"https://api.github.com/repos/polydawn/repeatr/events","assignees_url":"https://api.github.com/repos/polydawn/repeatr/assignees{/user}","branches_url":"https://api.github.com/repos/polydawn/repeatr/branches{/branch}","tags_url":"https://api.github.com/repos/polydawn/repeatr/tags","blobs_url":"https://api.github.com/repos/polydawn/repeatr/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/polydawn/repeatr/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/polydawn/repeatr/git/refs{/sha}","trees_url":"https://api.github.com/repos/polydawn/repeatr/git/trees{/sha}","statuses_url":"https://api.github.com/repos/polydawn/repeatr/statuses/{sha}","languages_url":"https://api.github.com/repos/polydawn/repeatr/languages","stargazers_url":"https://api.github.com/repos/polydawn/repeatr/stargazers","contributors_url":"https://api.github.com/repos/polydawn/repeatr/contributors","subscribers_url":"https://api.github.com/repos/polydawn/repeatr/subscribers","subscription_url":"https://api.github.com/repos/polydawn/repeatr/subscription","commits_url":"https://api.github.com/repos/polydawn/repeatr/commits{/sha}","git_commits_url":"https://api.github.com/repos/polydawn/repeatr/git/commits{/sha}","comments_url":"https://api.github.com/repos/polydawn/repeatr/comments{/number}","issue_comment_url":"https://api.github.com/repos/polydawn/repeatr/issues/comments{/number}","contents_url":"https://api.github.com/repos/polydawn/repeatr/contents/{+path}","compare_url":"https://api.github.com/repos/polydawn/repeatr/compare/{base}...{head}","merges_url":"https://api.github.com/repos/polydawn/repeatr/merges","archive_url":"https://api.github.com/repos/polydawn/repeatr/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/polydawn/repeatr/downloads","issues_url":"https://api.github.com/repos/polydawn/repeatr/issues{/number}","pulls_url":"https://api.github.com/repos/polydawn/repeatr/pulls{/number}","milestones_url":"https://api.github.com/repos/polydawn/repeatr/milestones{/number}","notifications_url":"https://api.github.com/repos/polydawn/repeatr/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/polydawn/repeatr/labels{/name}","releases_url":"https://api.github.com/repos/polydawn/repeatr/releases{/id}","created_at":"2015-02-27T18:12:08Z","updated_at":"2015-05-05T03:05:10Z","pushed_at":"2015-05-05T04:09:53Z","git_url":"git://github.com/polydawn/repeatr.git","ssh_url":"git@github.com:polydawn/repeatr.git","clone_url":"https://github.com/polydawn/repeatr.git","svn_url":"https://github.com/polydawn/repeatr","homepage":null,"size":2225,"stargazers_count":3,"watchers_count":3,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":3,"forks":1,"open_issues":3,"watchers":3,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/polydawn/repeatr/pulls/42"},"html":{"href":"https://github.com/polydawn/repeatr/pull/42"},"issue":{"href":"https://api.github.com/repos/polydawn/repeatr/issues/42"},"comments":{"href":"https://api.github.com/repos/polydawn/repeatr/issues/42/comments"},"review_comments":{"href":"https://api.github.com/repos/polydawn/repeatr/pulls/42/comments"},"review_comment":{"href":"https://api.github.com/repos/polydawn/repeatr/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/polydawn/repeatr/pulls/42/commits"},"statuses":{"href":"https://api.github.com/repos/polydawn/repeatr/statuses/b0a8d01e9708f2c90130ad8163ac75247ac14e96"}}}}