{"action":"created","issue":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/4552","labels_url":"https://api.github.com/repos/influxdb/influxdb/issues/4552/labels{/name}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/4552/comments","events_url":"https://api.github.com/repos/influxdb/influxdb/issues/4552/events","html_url":"https://github.com/influxdb/influxdb/issues/4552","id":113053179,"number":4552,"title":"0.9.5-nightly-827c513 process write shard error: write shard 4: engine: WAL backed up flushing to index, hit max memory","user":{"login":"wrigtim","id":1384495,"avatar_url":"https://avatars.githubusercontent.com/u/1384495?v=3","gravatar_id":"","url":"https://api.github.com/users/wrigtim","html_url":"https://github.com/wrigtim","followers_url":"https://api.github.com/users/wrigtim/followers","following_url":"https://api.github.com/users/wrigtim/following{/other_user}","gists_url":"https://api.github.com/users/wrigtim/gists{/gist_id}","starred_url":"https://api.github.com/users/wrigtim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wrigtim/subscriptions","organizations_url":"https://api.github.com/users/wrigtim/orgs","repos_url":"https://api.github.com/users/wrigtim/repos","events_url":"https://api.github.com/users/wrigtim/events{/privacy}","received_events_url":"https://api.github.com/users/wrigtim/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/influxdb/influxdb/labels/tsm","name":"tsm","color":"e11d21"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":24,"created_at":"2015-10-23T16:20:19Z","updated_at":"2015-11-02T02:20:06Z","closed_at":null,"body":"We upgraded to 0.9.5-nightly-827c513 today to take the fix for #4388. It was running fairly well for around 40 minutes (no errors in logs), until the node we are writing to in the 3-node cluster started reporting a large number of the below write failures. The other two nodes reported the same errors shortly afterwards.\r\n\r\nDirectory listing for telegraf/default/4/: https://gist.github.com/wrigtim/d0c086d589539dda0ace\r\n\r\nCurrently running TSM/WAL on PCIe SSDs to hopefully reduce IO performance impact...\r\n\r\n```\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[handoff] 2015/10/23 11:07:26 remote write failed: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n[write] 2015/10/23 11:07:26 write failed for shard 4 on node 2: error code 1: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n```"},"comment":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/comments/152892849","html_url":"https://github.com/influxdb/influxdb/issues/4552#issuecomment-152892849","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/4552","id":152892849,"user":{"login":"sebito91","id":3159454,"avatar_url":"https://avatars.githubusercontent.com/u/3159454?v=3","gravatar_id":"","url":"https://api.github.com/users/sebito91","html_url":"https://github.com/sebito91","followers_url":"https://api.github.com/users/sebito91/followers","following_url":"https://api.github.com/users/sebito91/following{/other_user}","gists_url":"https://api.github.com/users/sebito91/gists{/gist_id}","starred_url":"https://api.github.com/users/sebito91/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sebito91/subscriptions","organizations_url":"https://api.github.com/users/sebito91/orgs","repos_url":"https://api.github.com/users/sebito91/repos","events_url":"https://api.github.com/users/sebito91/events{/privacy}","received_events_url":"https://api.github.com/users/sebito91/received_events","type":"User","site_admin":false},"created_at":"2015-11-02T02:20:06Z","updated_at":"2015-11-02T02:20:06Z","body":"@otoolep, @pauldix ...any ideas on the graph we included a couple of days ago?\r\n\r\nWhat's interesting with this development is that the app stayed up (no panic), but writes were blocked for a long period while write_requests continued to grow...so something is still listening for connections but unable to commit fast enough to the wal/index."}}