{"action":"created","issue":{"url":"https://api.github.com/repos/hashicorp/consul/issues/1062","labels_url":"https://api.github.com/repos/hashicorp/consul/issues/1062/labels{/name}","comments_url":"https://api.github.com/repos/hashicorp/consul/issues/1062/comments","events_url":"https://api.github.com/repos/hashicorp/consul/issues/1062/events","html_url":"https://github.com/hashicorp/consul/issues/1062","id":91258834,"number":1062,"title":"Intent queue is constantly overflowing ","user":{"login":"pchojnacki","id":812891,"avatar_url":"https://avatars.githubusercontent.com/u/812891?v=3","gravatar_id":"","url":"https://api.github.com/users/pchojnacki","html_url":"https://github.com/pchojnacki","followers_url":"https://api.github.com/users/pchojnacki/followers","following_url":"https://api.github.com/users/pchojnacki/following{/other_user}","gists_url":"https://api.github.com/users/pchojnacki/gists{/gist_id}","starred_url":"https://api.github.com/users/pchojnacki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pchojnacki/subscriptions","organizations_url":"https://api.github.com/users/pchojnacki/orgs","repos_url":"https://api.github.com/users/pchojnacki/repos","events_url":"https://api.github.com/users/pchojnacki/events{/privacy}","received_events_url":"https://api.github.com/users/pchojnacki/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"slackpad","id":673509,"avatar_url":"https://avatars.githubusercontent.com/u/673509?v=3","gravatar_id":"","url":"https://api.github.com/users/slackpad","html_url":"https://github.com/slackpad","followers_url":"https://api.github.com/users/slackpad/followers","following_url":"https://api.github.com/users/slackpad/following{/other_user}","gists_url":"https://api.github.com/users/slackpad/gists{/gist_id}","starred_url":"https://api.github.com/users/slackpad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/slackpad/subscriptions","organizations_url":"https://api.github.com/users/slackpad/orgs","repos_url":"https://api.github.com/users/slackpad/repos","events_url":"https://api.github.com/users/slackpad/events{/privacy}","received_events_url":"https://api.github.com/users/slackpad/received_events","type":"User","site_admin":false},"milestone":null,"comments":11,"created_at":"2015-06-26T13:58:48Z","updated_at":"2015-08-31T21:43:37Z","closed_at":null,"body":"In our setup that was recently upgraded to 0.5.2 we currently have two different Consul DCs/clusters configured in one physical Datacenter.\r\nOne is 'production' only hosts that have better monitoring and are meant to serve production traffic. The second is meant for development only.\r\n\r\nBefore migrating to 0.5.2 we also experienced problems described here:\r\nhttps://github.com/hashicorp/consul/issues/843\r\n\r\nNow the main problem is that we receive tons of logs like these from the dev cluster:\r\n```\r\n    2015/06/26 13:28:39 [WARN] serf: Intent queue depth: 4256\r\n    2015/06/26 13:28:39 [WARN] serf: Intent queue depth (4256) exceeds limit (4096), dropping messages!\r\n```\r\nEssentially every agent is broadcasting log lines similar to almost every second.\r\nWhile cluster seems fine we are worried that some performance might be degraded. \r\n\r\nTo debug this issue I've run one of the agents in debug mode. Where I get tons of messageJoinType interspersed with warnings about queue depth. The nodes mentioned in messageJoinType are *always* from different cluster.\r\n\r\nE.g.\r\n```\r\n   2015/06/26 13:27:08 [DEBUG] serf: messageJoinType:  XXXXXXse3\r\n    2015/06/26 13:27:08 [DEBUG] serf: messageJoinType:  XXXXXX-s2\r\n    2015/06/26 13:27:08 [DEBUG] serf: messageJoinType:  XXXXXX-s2\r\n    2015/06/26 13:27:08 [DEBUG] serf: messageJoinType:  XXXXXXsc3\r\n    2015/06/26 13:27:08 [WARN] serf: Intent queue depth: 2313\r\n    2015/06/26 13:27:08 [DEBUG] serf: messageJoinType:  XXXXXXrev\r\n    2015/06/26 13:27:08 [DEBUG] serf: messageJoinType:  XXXXXXsf1\r\n    2015/06/26 13:27:08 [DEBUG] serf: messageJoinType:  XXXXXXsc5\r\n    2015/06/26 13:27:08 [DEBUG] serf: messageJoinType:  XXXXXX-de\r\n```\r\nand\r\n```\r\n\r\n    2015/06/26 13:28:39 [DEBUG] serf: messageJoinType:  XXXXXX-s1\r\n    2015/06/26 13:28:39 [DEBUG] serf: messageJoinType:  XXXXXXs10\r\n    2015/06/26 13:28:39 [DEBUG] serf: messageJoinType:  XXXXXXsc3\r\n    2015/06/26 13:28:39 [WARN] serf: Intent queue depth: 4256\r\n    2015/06/26 13:28:39 [WARN] serf: Intent queue depth (4256) exceeds limit (4096), dropping messages!\r\n    2015/06/26 13:28:39 [DEBUG] serf: messageJoinType:  XXXXXX-s7\r\n    2015/06/26 13:28:39 [DEBUG] serf: messageJoinType:  XXXXXXa12\r\n    2015/06/26 13:28:39 [DEBUG] serf: messageJoinType:  XXXXXXs13\r\n```\r\n\r\nI've also tried latest master branch Consul  and the result is the same.\r\nI haven't tried full cluster restarts yet with cleaning up data files etc. And preferably I'd like to avoid that.\r\n"},"comment":{"url":"https://api.github.com/repos/hashicorp/consul/issues/comments/136509799","html_url":"https://github.com/hashicorp/consul/issues/1062#issuecomment-136509799","issue_url":"https://api.github.com/repos/hashicorp/consul/issues/1062","id":136509799,"user":{"login":"armon","id":592032,"avatar_url":"https://avatars.githubusercontent.com/u/592032?v=3","gravatar_id":"","url":"https://api.github.com/users/armon","html_url":"https://github.com/armon","followers_url":"https://api.github.com/users/armon/followers","following_url":"https://api.github.com/users/armon/following{/other_user}","gists_url":"https://api.github.com/users/armon/gists{/gist_id}","starred_url":"https://api.github.com/users/armon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/armon/subscriptions","organizations_url":"https://api.github.com/users/armon/orgs","repos_url":"https://api.github.com/users/armon/repos","events_url":"https://api.github.com/users/armon/events{/privacy}","received_events_url":"https://api.github.com/users/armon/received_events","type":"User","site_admin":false},"created_at":"2015-08-31T21:43:37Z","updated_at":"2015-08-31T21:43:37Z","body":"@jaredcurtis Probably the best short term fix is a custom build that sets the intent buffer to a much larger size. That is a bit of a hack, and we need to think through a longer term solution, but there isn't an obvious answer."}}