{"action":"created","issue":{"url":"https://api.github.com/repos/awslabs/amazon-elasticsearch-lambda-samples/issues/1","labels_url":"https://api.github.com/repos/awslabs/amazon-elasticsearch-lambda-samples/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/awslabs/amazon-elasticsearch-lambda-samples/issues/1/comments","events_url":"https://api.github.com/repos/awslabs/amazon-elasticsearch-lambda-samples/issues/1/events","html_url":"https://github.com/awslabs/amazon-elasticsearch-lambda-samples/issues/1","id":114748238,"number":1,"title":"kinesis -> lambda -> ES only posting some records","user":{"login":"csakoda","id":4173737,"avatar_url":"https://avatars.githubusercontent.com/u/4173737?v=3","gravatar_id":"","url":"https://api.github.com/users/csakoda","html_url":"https://github.com/csakoda","followers_url":"https://api.github.com/users/csakoda/followers","following_url":"https://api.github.com/users/csakoda/following{/other_user}","gists_url":"https://api.github.com/users/csakoda/gists{/gist_id}","starred_url":"https://api.github.com/users/csakoda/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/csakoda/subscriptions","organizations_url":"https://api.github.com/users/csakoda/orgs","repos_url":"https://api.github.com/users/csakoda/repos","events_url":"https://api.github.com/users/csakoda/events{/privacy}","received_events_url":"https://api.github.com/users/csakoda/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-11-03T06:01:25Z","updated_at":"2015-11-04T10:07:49Z","closed_at":null,"body":"I setup a simple config to test this out:\r\n\r\nlogstash-forwarder -> logstash -> kinesis -> lambda -> ES\r\n\r\nFor batches of size 1, this works fine, log lines are pushed to ES in a few seconds.\r\n\r\nFor batch size > 1, I am seeing inconsistent behavior, with one of some of my records being pushed.  It's often about 5 records that make it through but it's not consistent.  \r\n\r\nIn my lambda logs I sometimes see `Error: Error: socket hang up` a number of times, corresponding to the number of missing records, but this too is inconsistent.\r\n\r\nI've confirmed that the expected # of the records appear in the kinesis stream and are transmitted to the lambda job. \r\n\r\nWhat am I doing wrong?  Is my kinesis stream too small?  My lambda timeout too short?"},"comment":{"url":"https://api.github.com/repos/awslabs/amazon-elasticsearch-lambda-samples/issues/comments/153668609","html_url":"https://github.com/awslabs/amazon-elasticsearch-lambda-samples/issues/1#issuecomment-153668609","issue_url":"https://api.github.com/repos/awslabs/amazon-elasticsearch-lambda-samples/issues/1","id":153668609,"user":{"login":"srisub-amzn","id":14072155,"avatar_url":"https://avatars.githubusercontent.com/u/14072155?v=3","gravatar_id":"","url":"https://api.github.com/users/srisub-amzn","html_url":"https://github.com/srisub-amzn","followers_url":"https://api.github.com/users/srisub-amzn/followers","following_url":"https://api.github.com/users/srisub-amzn/following{/other_user}","gists_url":"https://api.github.com/users/srisub-amzn/gists{/gist_id}","starred_url":"https://api.github.com/users/srisub-amzn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/srisub-amzn/subscriptions","organizations_url":"https://api.github.com/users/srisub-amzn/orgs","repos_url":"https://api.github.com/users/srisub-amzn/repos","events_url":"https://api.github.com/users/srisub-amzn/events{/privacy}","received_events_url":"https://api.github.com/users/srisub-amzn/received_events","type":"User","site_admin":false},"created_at":"2015-11-04T10:07:49Z","updated_at":"2015-11-04T10:07:49Z","body":"Thanks for trying out the sample!\r\n\r\nThe code is meant to demonstrate how Lambda could be used for ES data ingestion. It is simplified for the purposes of clarity and is not tuned to work at scale."}}