{"action":"created","issue":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-s3/issues/14","labels_url":"https://api.github.com/repos/logstash-plugins/logstash-input-s3/issues/14/labels{/name}","comments_url":"https://api.github.com/repos/logstash-plugins/logstash-input-s3/issues/14/comments","events_url":"https://api.github.com/repos/logstash-plugins/logstash-input-s3/issues/14/events","html_url":"https://github.com/logstash-plugins/logstash-input-s3/issues/14","id":56886531,"number":14,"title":"S3 input and large buckets","user":{"login":"talevy","id":388837,"avatar_url":"https://avatars.githubusercontent.com/u/388837?v=3","gravatar_id":"","url":"https://api.github.com/users/talevy","html_url":"https://github.com/talevy","followers_url":"https://api.github.com/users/talevy/followers","following_url":"https://api.github.com/users/talevy/following{/other_user}","gists_url":"https://api.github.com/users/talevy/gists{/gist_id}","starred_url":"https://api.github.com/users/talevy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talevy/subscriptions","organizations_url":"https://api.github.com/users/talevy/orgs","repos_url":"https://api.github.com/users/talevy/repos","events_url":"https://api.github.com/users/talevy/events{/privacy}","received_events_url":"https://api.github.com/users/talevy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-02-07T00:14:49Z","updated_at":"2015-03-10T17:36:41Z","closed_at":null,"body":"migrated from: https://logstash.jira.com/browse/LOGSTASH-2125\r\n\r\nS3 input is takeing a long time until the first logfile is processed:\r\n\r\n```\r\ninput {\r\n    s3 {\r\n        credentials => [\"XXXX\",\"XXXX\"]\r\n        bucket => \"my-production-bucket\"\r\n        interval => 300\r\n   }\r\n}\r\noutput {\r\n   stdout {}\r\n}\r\n```\r\n\r\nRunning it with\r\n```\r\nsudo ./logstash agent -f /etc/logstash/conf.d/central.conf  --debug\r\n```\r\n\r\nshows me that the bucket is used. As soon as I start logstash I see via tcpdump that there is a lot of traffic between the host and s3 going on.\r\nNow that bucket has right now 4451 .gz files just in the root folder. Subfolders have even more files.\r\nIf I create now another bucket and put only one of the log files in it I can see that this logfile is more or less immediately downloaded and processed."},"comment":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-s3/issues/comments/78106010","html_url":"https://github.com/logstash-plugins/logstash-input-s3/issues/14#issuecomment-78106010","issue_url":"https://api.github.com/repos/logstash-plugins/logstash-input-s3/issues/14","id":78106010,"user":{"login":"DanielRedOak","id":5090638,"avatar_url":"https://avatars.githubusercontent.com/u/5090638?v=3","gravatar_id":"","url":"https://api.github.com/users/DanielRedOak","html_url":"https://github.com/DanielRedOak","followers_url":"https://api.github.com/users/DanielRedOak/followers","following_url":"https://api.github.com/users/DanielRedOak/following{/other_user}","gists_url":"https://api.github.com/users/DanielRedOak/gists{/gist_id}","starred_url":"https://api.github.com/users/DanielRedOak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DanielRedOak/subscriptions","organizations_url":"https://api.github.com/users/DanielRedOak/orgs","repos_url":"https://api.github.com/users/DanielRedOak/repos","events_url":"https://api.github.com/users/DanielRedOak/events{/privacy}","received_events_url":"https://api.github.com/users/DanielRedOak/received_events","type":"User","site_admin":false},"created_at":"2015-03-10T17:36:41Z","updated_at":"2015-03-10T17:36:41Z","body":"This is related I believe: https://github.com/aws/aws-sdk-ruby/issues/588\r\n\r\nSo since this uses aws-sdk < 2 I think we're SOL until its upgraded.  I'll see about it if I have some time here, but there is a ticket open to get the mixin updated too"}}