{"action":"created","issue":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-s3/issues/25","labels_url":"https://api.github.com/repos/logstash-plugins/logstash-input-s3/issues/25/labels{/name}","comments_url":"https://api.github.com/repos/logstash-plugins/logstash-input-s3/issues/25/comments","events_url":"https://api.github.com/repos/logstash-plugins/logstash-input-s3/issues/25/events","html_url":"https://github.com/logstash-plugins/logstash-input-s3/pull/25","id":60698682,"number":25,"title":"Upgrade to aws-sdk v2 to get feature improvements","user":{"login":"DanielRedOak","id":5090638,"avatar_url":"https://avatars.githubusercontent.com/u/5090638?v=3","gravatar_id":"","url":"https://api.github.com/users/DanielRedOak","html_url":"https://github.com/DanielRedOak","followers_url":"https://api.github.com/users/DanielRedOak/followers","following_url":"https://api.github.com/users/DanielRedOak/following{/other_user}","gists_url":"https://api.github.com/users/DanielRedOak/gists{/gist_id}","starred_url":"https://api.github.com/users/DanielRedOak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DanielRedOak/subscriptions","organizations_url":"https://api.github.com/users/DanielRedOak/orgs","repos_url":"https://api.github.com/users/DanielRedOak/repos","events_url":"https://api.github.com/users/DanielRedOak/events{/privacy}","received_events_url":"https://api.github.com/users/DanielRedOak/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-s3/labels/missing%20cla","name":"missing cla","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":21,"created_at":"2015-03-11T17:46:21Z","updated_at":"2015-12-17T13:49:59Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-s3/pulls/25","html_url":"https://github.com/logstash-plugins/logstash-input-s3/pull/25","diff_url":"https://github.com/logstash-plugins/logstash-input-s3/pull/25.diff","patch_url":"https://github.com/logstash-plugins/logstash-input-s3/pull/25.patch"},"body":"Upgraded spec tests + code to use aws-sdk v2.  This then improves processing by a large margin as v1 repeated api calls for every object when .last_modified was used.  v2 requires only a single call for every 1000 objects so on a bucket with 1000 objects, api calls are down to 1 from 1001 just for the last_modified and bucket.objects calls.  Pagination is included in the sdk as well so this plugin now supports buckets with over 1000 objects.\r\n\r\nAdditionally added a debug message to show the length of objects[] as things are being added.\r\n\r\nThere was also a bug where the 'message' field was assumed present and used for checks to determine if the 'event_is_metadata'.  Since this is not a required field per https://logstash.jira.com/browse/LOGSTASH-675 we should check for nil before assuming its present.  This manifested itself specifically when using the cloudtrail codec as there is no message included in resulting events."},"comment":{"url":"https://api.github.com/repos/logstash-plugins/logstash-input-s3/issues/comments/165457559","html_url":"https://github.com/logstash-plugins/logstash-input-s3/pull/25#issuecomment-165457559","issue_url":"https://api.github.com/repos/logstash-plugins/logstash-input-s3/issues/25","id":165457559,"user":{"login":"KptnKMan","id":12234043,"avatar_url":"https://avatars.githubusercontent.com/u/12234043?v=3","gravatar_id":"","url":"https://api.github.com/users/KptnKMan","html_url":"https://github.com/KptnKMan","followers_url":"https://api.github.com/users/KptnKMan/followers","following_url":"https://api.github.com/users/KptnKMan/following{/other_user}","gists_url":"https://api.github.com/users/KptnKMan/gists{/gist_id}","starred_url":"https://api.github.com/users/KptnKMan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KptnKMan/subscriptions","organizations_url":"https://api.github.com/users/KptnKMan/orgs","repos_url":"https://api.github.com/users/KptnKMan/repos","events_url":"https://api.github.com/users/KptnKMan/events{/privacy}","received_events_url":"https://api.github.com/users/KptnKMan/received_events","type":"User","site_admin":false},"created_at":"2015-12-17T13:49:59Z","updated_at":"2015-12-17T13:49:59Z","body":"Omg @sstarcher, thank you! Is there any special compiling required for this\nor can I just clone the repository and use the files?\n\nI'm happy to test!\nOn 17 Dec 2015 14:33, \"Shane Starcher\" <notifications@github.com> wrote:\n\n> @smashew <https://github.com/smashew> if your interested I rebased\n> @DanielRedOak <https://github.com/DanielRedOak>\n> https://github.com/DanielRedOak/logstash-input-s3/tree/multithread\n> changes for aws-sdk v2 and his multithreading changes onto\n> logstash-input-s3 HEAD onto\n> https://github.com/sstarcher/logstash-input-s3/\n>\n> I have been running it for a few days now. It is performing much much\n> better. I can't seem to get it to run more than on 3 cores though so it's\n> still limited. Although that may be a user error.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/logstash-plugins/logstash-input-s3/pull/25#issuecomment-165454331>\n> .\n>\n"}}