{"action":"created","issue":{"url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/1561","labels_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/1561/labels{/name}","comments_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/1561/comments","events_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/1561/events","html_url":"https://github.com/Graylog2/graylog2-server/issues/1561","id":116697155,"number":1561,"title":"Probable concurrency bug in GelfChunkAggregator.checkForCompletion ()","user":{"login":"uttg","id":960862,"avatar_url":"https://avatars.githubusercontent.com/u/960862?v=3","gravatar_id":"","url":"https://api.github.com/users/uttg","html_url":"https://github.com/uttg","followers_url":"https://api.github.com/users/uttg/followers","following_url":"https://api.github.com/users/uttg/following{/other_user}","gists_url":"https://api.github.com/users/uttg/gists{/gist_id}","starred_url":"https://api.github.com/users/uttg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/uttg/subscriptions","organizations_url":"https://api.github.com/users/uttg/orgs","repos_url":"https://api.github.com/users/uttg/repos","events_url":"https://api.github.com/users/uttg/events{/privacy}","received_events_url":"https://api.github.com/users/uttg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-11-13T04:16:07Z","updated_at":"2015-11-19T14:57:46Z","closed_at":null,"body":"Hello,\r\n\r\nI investigated on #1544 a little bit further.\r\n\r\nFor a message with 6 chunks I have the following behaviour:\r\n* *reception of the first chunks...*\r\n* reception of chunk number 3\r\n  * checkForCompletion detects that we have every chunks\r\n  * it fails to construct a full message. Chunks 4, 5 and 6 are not yet in entry.payloadArray\r\n* reception of chunk number 6\r\n  * checkForCompletion detects that we have every chunks\r\n  * it fails to construct a full message. Chunks 1, 2 and 3 are not yet in entry.payloadArray\r\n\r\nI noticed that the test [if (chunkWatermark == sequenceCount)](https://github.com/Graylog2/graylog2-server/blob/99d3e248090547114cb8d5b0ad4f07cf9abbde27/graylog2-inputs/src/main/java/org/graylog2/inputs/codecs/GelfChunkAggregator.java#L144) may return true multiple times for the same message.\r\n\r\nSomehow, [incrementAndGet()](https://github.com/Graylog2/graylog2-server/blob/99d3e248090547114cb8d5b0ad4f07cf9abbde27/graylog2-inputs/src/main/java/org/graylog2/inputs/codecs/GelfChunkAggregator.java#L141) returns the same value multiple times event though the operation should be atomic.\r\n\r\nIn graylog.conf we have:\r\n```\r\nprocessor_wait_strategy = blocking\r\ninputbuffer_processors = 2\r\ninputbuffer_wait_strategy = blocking\r\n```"},"comment":{"url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/comments/158081377","html_url":"https://github.com/Graylog2/graylog2-server/issues/1561#issuecomment-158081377","issue_url":"https://api.github.com/repos/Graylog2/graylog2-server/issues/1561","id":158081377,"user":{"login":"joschi","id":43951,"avatar_url":"https://avatars.githubusercontent.com/u/43951?v=3","gravatar_id":"","url":"https://api.github.com/users/joschi","html_url":"https://github.com/joschi","followers_url":"https://api.github.com/users/joschi/followers","following_url":"https://api.github.com/users/joschi/following{/other_user}","gists_url":"https://api.github.com/users/joschi/gists{/gist_id}","starred_url":"https://api.github.com/users/joschi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joschi/subscriptions","organizations_url":"https://api.github.com/users/joschi/orgs","repos_url":"https://api.github.com/users/joschi/repos","events_url":"https://api.github.com/users/joschi/events{/privacy}","received_events_url":"https://api.github.com/users/joschi/received_events","type":"User","site_admin":false},"created_at":"2015-11-19T14:57:46Z","updated_at":"2015-11-19T14:57:46Z","body":"> I'm not overly sure about the error log level, because that can easily flood the server log, too.\r\n\r\nThis error *should not* happen often and if it does it's a sign of a broken client, so I think it's ok to flood the logs with those messages if this error occurs.\r\n\r\n> At the very least we should add an additional counter metric, I think.\r\n\r\nYes."}}