{"action":"created","issue":{"url":"https://api.github.com/repos/smira/aptly/issues/297","labels_url":"https://api.github.com/repos/smira/aptly/issues/297/labels{/name}","comments_url":"https://api.github.com/repos/smira/aptly/issues/297/comments","events_url":"https://api.github.com/repos/smira/aptly/issues/297/events","html_url":"https://github.com/smira/aptly/issues/297","id":104930184,"number":297,"title":"S3 Publishing Large Repos","user":{"login":"mattford63","id":400074,"avatar_url":"https://avatars.githubusercontent.com/u/400074?v=3","gravatar_id":"","url":"https://api.github.com/users/mattford63","html_url":"https://github.com/mattford63","followers_url":"https://api.github.com/users/mattford63/followers","following_url":"https://api.github.com/users/mattford63/following{/other_user}","gists_url":"https://api.github.com/users/mattford63/gists{/gist_id}","starred_url":"https://api.github.com/users/mattford63/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattford63/subscriptions","organizations_url":"https://api.github.com/users/mattford63/orgs","repos_url":"https://api.github.com/users/mattford63/repos","events_url":"https://api.github.com/users/mattford63/events{/privacy}","received_events_url":"https://api.github.com/users/mattford63/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/smira/aptly/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/smira/aptly/milestones/13","html_url":"https://github.com/smira/aptly/milestones/v0.9.6","labels_url":"https://api.github.com/repos/smira/aptly/milestones/13/labels","id":1020064,"number":13,"title":"v0.9.6","description":"","creator":{"login":"smira","id":55211,"avatar_url":"https://avatars.githubusercontent.com/u/55211?v=3","gravatar_id":"","url":"https://api.github.com/users/smira","html_url":"https://github.com/smira","followers_url":"https://api.github.com/users/smira/followers","following_url":"https://api.github.com/users/smira/following{/other_user}","gists_url":"https://api.github.com/users/smira/gists{/gist_id}","starred_url":"https://api.github.com/users/smira/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smira/subscriptions","organizations_url":"https://api.github.com/users/smira/orgs","repos_url":"https://api.github.com/users/smira/repos","events_url":"https://api.github.com/users/smira/events{/privacy}","received_events_url":"https://api.github.com/users/smira/received_events","type":"User","site_admin":false},"open_issues":20,"closed_issues":20,"state":"open","created_at":"2015-03-13T10:24:41Z","updated_at":"2015-10-01T09:40:59Z","due_on":null,"closed_at":null},"comments":13,"created_at":"2015-09-04T17:06:01Z","updated_at":"2015-10-03T02:06:03Z","closed_at":null,"body":"When publishing large mirrors to S3 I quickly hit the S3 API throttling limits.\r\n\r\nAptly doesn't seem to gracefully scale down (boto based stuff does) and simply fails the publish instead.\r\n\r\nMy working theory is the many HEAD requests to check file existence happen quickly enough to cause issues.\r\n\r\nTo re-create try to upload, say, trusty-updates to s3.\r\n\r\nI've tried rate limiting with IPTables and the user space throttler trickle.  Too not much success.  I'd love to see if others have seen this issue?"},"comment":{"url":"https://api.github.com/repos/smira/aptly/issues/comments/145193495","html_url":"https://github.com/smira/aptly/issues/297#issuecomment-145193495","issue_url":"https://api.github.com/repos/smira/aptly/issues/297","id":145193495,"user":{"login":"mdxp","id":84507,"avatar_url":"https://avatars.githubusercontent.com/u/84507?v=3","gravatar_id":"","url":"https://api.github.com/users/mdxp","html_url":"https://github.com/mdxp","followers_url":"https://api.github.com/users/mdxp/followers","following_url":"https://api.github.com/users/mdxp/following{/other_user}","gists_url":"https://api.github.com/users/mdxp/gists{/gist_id}","starred_url":"https://api.github.com/users/mdxp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdxp/subscriptions","organizations_url":"https://api.github.com/users/mdxp/orgs","repos_url":"https://api.github.com/users/mdxp/repos","events_url":"https://api.github.com/users/mdxp/events{/privacy}","received_events_url":"https://api.github.com/users/mdxp/received_events","type":"User","site_admin":false},"created_at":"2015-10-03T02:06:03Z","updated_at":"2015-10-03T02:06:03Z","body":"OMG @smira this is awesome. I just tried it out and it is amazingly fast; I'm sure that anyone that's using aptly with S3 will love this. I tested various scenarios and it worked out great. Unrelated from this I've found a bug (that was always there from what I can tell), the 'file exist' check will not verify for `plusWorkaround` type of packages (if enabled), and just ignore those.  "}}