{"action":"created","issue":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/4562","labels_url":"https://api.github.com/repos/influxdb/influxdb/issues/4562/labels{/name}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/4562/comments","events_url":"https://api.github.com/repos/influxdb/influxdb/issues/4562/events","html_url":"https://github.com/influxdb/influxdb/issues/4562","id":113151579,"number":4562,"title":"0.9.5-nightly-9633410 Telegraf isnt able to write","user":{"login":"daviesalex","id":1106249,"avatar_url":"https://avatars.githubusercontent.com/u/1106249?v=3","gravatar_id":"","url":"https://api.github.com/users/daviesalex","html_url":"https://github.com/daviesalex","followers_url":"https://api.github.com/users/daviesalex/followers","following_url":"https://api.github.com/users/daviesalex/following{/other_user}","gists_url":"https://api.github.com/users/daviesalex/gists{/gist_id}","starred_url":"https://api.github.com/users/daviesalex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/daviesalex/subscriptions","organizations_url":"https://api.github.com/users/daviesalex/orgs","repos_url":"https://api.github.com/users/daviesalex/repos","events_url":"https://api.github.com/users/daviesalex/events{/privacy}","received_events_url":"https://api.github.com/users/daviesalex/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/influxdb/influxdb/labels/tsm","name":"tsm","color":"e11d21"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-10-24T11:16:42Z","updated_at":"2015-11-05T11:26:55Z","closed_at":"2015-11-05T11:26:55Z","body":"If I delete all my data, and start up a 3 node cluster, the writer returns a 404 to every request:\r\n\r\n```\r\n[http] 2015/10/24 05:46:12 7.132.32.31 - - [24/Oct/2015:05:46:12 -0500] POST /write?consistency=&db=telegraf&precision=ns&q=CREATE+DATABASE+telegraf&rp= HTTP/1.1 404 55 - InfluxDBClient 7185f836-7a3c-11e5-83e1-000000000000 255.131694ms\r\n[http] 2015/10/24 05:46:12 7.132.32.31 - - [24/Oct/2015:05:46:12 -0500] POST /write?consistency=&db=telegraf&precision=ns&q=CREATE+DATABASE+telegraf&rp= HTTP/1.1 404 55 - InfluxDBClient 7137fb28-7a3c-11e5-8383-000000000000 766.649046ms\r\n[http] 2015/10/24 05:46:12 7.132.32.31 - - [24/Oct/2015:05:46:12 -0500] POST /write?consistency=&db=telegraf&precision=ns&q=CREATE+DATABASE+telegraf&rp= HTTP/1.1 404 55 - InfluxDBClient 71a71b5e-7a3c-11e5-83fe-000000000000 39.980522ms\r\n```\r\n\r\nClearly, telegraf is attempting to create the database and failing. However, as soon as I run \"CREATE DATABASE telegraf\" from the command line on the same node, the command suceeded and these HTTP codes change to HTTP 500:\r\n\r\n```\r\n[http] 2015/10/24 06:13:11 7.132.32.31 - - [24/Oct/2015:06:12:50 -0500] POST /write?consistency=&db=telegraf&precision=ns&q=CREATE+DATABASE+telegraf&rp= HTTP/1.1 500 32 - InfluxDBClient 2a15e307-7a40-11e5-82b0-000000000000 20.880270601s\r\n[http] 2015/10/24 06:13:11 7.132.32.31 - - [24/Oct/2015:06:12:44 -0500] POST /write?consistency=&db=telegraf&precision=ns&q=CREATE+DATABASE+telegraf&rp= HTTP/1.1 500 32 - InfluxDBClient 2689316a-7a40-11e5-80b4-000000000000 26.835959577s\r\n[http] 2015/10/24 06:13:11 7.132.32.31 - - [24/Oct/2015:06:12:49 -0500] POST /write?consistency=&db=telegraf&precision=ns&q=CREATE+DATABASE+telegraf&rp= HTTP/1.1 500 32 - InfluxDBClient 29272100-7a40-11e5-81ac-000000000000 22.445715559s\r\n```\r\n\r\nThere are virtually no \"err\" or \"panic\" messages in the log/stdout.\r\n\r\nAfter about 15 minutes, we see hundreds of thousands of errors on the other 2 nodes:\r\n\r\n```\r\n[tcp] 2015/10/24 06:15:15 process write shard error: write shard 3: engine: WAL backed up flushing to index, hit max memory\r\n[tcp] 2015/10/24 06:15:15 process write shard error: write shard 3: engine: WAL backed up flushing to index, hit max memory\r\n[tcp] 2015/10/24 06:15:15 process write shard error: write shard 3: engine: WAL backed up flushing to index, hit max memory\r\n[tcp] 2015/10/24 06:15:14 process write shard error: write shard 3: engine: WAL backed up flushing to index, hit max memory\r\n[tcp] 2015/10/24 06:15:14 process write shard error: write shard 3: engine: WAL backed up flushing to index, hit max memory\r\n```\r\n\r\nThere is virtually no disk IO going on."},"comment":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/comments/154035743","html_url":"https://github.com/influxdb/influxdb/issues/4562#issuecomment-154035743","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/4562","id":154035743,"user":{"login":"daviesalex","id":1106249,"avatar_url":"https://avatars.githubusercontent.com/u/1106249?v=3","gravatar_id":"","url":"https://api.github.com/users/daviesalex","html_url":"https://github.com/daviesalex","followers_url":"https://api.github.com/users/daviesalex/followers","following_url":"https://api.github.com/users/daviesalex/following{/other_user}","gists_url":"https://api.github.com/users/daviesalex/gists{/gist_id}","starred_url":"https://api.github.com/users/daviesalex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/daviesalex/subscriptions","organizations_url":"https://api.github.com/users/daviesalex/orgs","repos_url":"https://api.github.com/users/daviesalex/repos","events_url":"https://api.github.com/users/daviesalex/events{/privacy}","received_events_url":"https://api.github.com/users/daviesalex/received_events","type":"User","site_admin":false},"created_at":"2015-11-05T11:26:55Z","updated_at":"2015-11-05T11:26:55Z","body":"I think we should close this as a duplicate of https://github.com/influxdb/influxdb/issues/4498\r\n\r\nHowever, this is clearly a problem with Influx, and not a load issue - we are seeing this consistently after a number of hours of running despite there being virtually no IO going on, and a huge amount of CPU time spare. Somehow this is getting stuck.\r\n\r\nWe see this after hours of uptime even with massively different numbers of machines connected. "}}