{"action":"created","issue":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/4498","labels_url":"https://api.github.com/repos/influxdb/influxdb/issues/4498/labels{/name}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/4498/comments","events_url":"https://api.github.com/repos/influxdb/influxdb/issues/4498/events","html_url":"https://github.com/influxdb/influxdb/issues/4498","id":112176370,"number":4498,"title":"0.9.5-nightly-cc54cba with tsm1 ignores wal-dir","user":{"login":"daviesalex","id":1106249,"avatar_url":"https://avatars.githubusercontent.com/u/1106249?v=3","gravatar_id":"","url":"https://api.github.com/users/daviesalex","html_url":"https://github.com/daviesalex","followers_url":"https://api.github.com/users/daviesalex/followers","following_url":"https://api.github.com/users/daviesalex/following{/other_user}","gists_url":"https://api.github.com/users/daviesalex/gists{/gist_id}","starred_url":"https://api.github.com/users/daviesalex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/daviesalex/subscriptions","organizations_url":"https://api.github.com/users/daviesalex/orgs","repos_url":"https://api.github.com/users/daviesalex/repos","events_url":"https://api.github.com/users/daviesalex/events{/privacy}","received_events_url":"https://api.github.com/users/daviesalex/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/influxdb/influxdb/labels/tsm","name":"tsm","color":"e11d21"}],"state":"open","locked":false,"assignee":{"login":"pauldix","id":4331,"avatar_url":"https://avatars.githubusercontent.com/u/4331?v=3","gravatar_id":"","url":"https://api.github.com/users/pauldix","html_url":"https://github.com/pauldix","followers_url":"https://api.github.com/users/pauldix/followers","following_url":"https://api.github.com/users/pauldix/following{/other_user}","gists_url":"https://api.github.com/users/pauldix/gists{/gist_id}","starred_url":"https://api.github.com/users/pauldix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pauldix/subscriptions","organizations_url":"https://api.github.com/users/pauldix/orgs","repos_url":"https://api.github.com/users/pauldix/repos","events_url":"https://api.github.com/users/pauldix/events{/privacy}","received_events_url":"https://api.github.com/users/pauldix/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/influxdb/influxdb/milestones/44","html_url":"https://github.com/influxdb/influxdb/milestones/0.9.5","labels_url":"https://api.github.com/repos/influxdb/influxdb/milestones/44/labels","id":1276659,"number":44,"title":"0.9.5","description":"","creator":{"login":"beckettsean","id":6537329,"avatar_url":"https://avatars.githubusercontent.com/u/6537329?v=3","gravatar_id":"","url":"https://api.github.com/users/beckettsean","html_url":"https://github.com/beckettsean","followers_url":"https://api.github.com/users/beckettsean/followers","following_url":"https://api.github.com/users/beckettsean/following{/other_user}","gists_url":"https://api.github.com/users/beckettsean/gists{/gist_id}","starred_url":"https://api.github.com/users/beckettsean/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beckettsean/subscriptions","organizations_url":"https://api.github.com/users/beckettsean/orgs","repos_url":"https://api.github.com/users/beckettsean/repos","events_url":"https://api.github.com/users/beckettsean/events{/privacy}","received_events_url":"https://api.github.com/users/beckettsean/received_events","type":"User","site_admin":false},"open_issues":37,"closed_issues":43,"state":"open","created_at":"2015-08-27T21:45:59Z","updated_at":"2015-10-22T12:42:32Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2015-10-19T16:00:39Z","updated_at":"2015-10-23T16:41:40Z","closed_at":null,"body":"The WAL design results in a very high number of fsync calls, which even with SAS SSDs is difficult to scale. We want to route WAL requests to a small NVMe volume, but it appears that when using tsm1 InfluxDB is creating the specified WAL directory but then not using it.\r\n\r\nThe WAL files are all all like this:\r\n\r\n```\r\n[root@host data]# lsof | grep wal  | head -2\r\ninfluxd   30764       influxdb    7u      REG              253,4     479453  249036904 /data/influxdb-data/metrics/telegraf/default/4/_02241.wal\r\ninfluxd   30764       influxdb   29u      REG              253,4       1484  249036843 /data/influxdb-data/metrics/_internal/monitor/3/_00040.wal\r\n```\r\n\r\nThere are in fact no open file handlers to the WAL dir:\r\n```\r\n[root@host data]# lsof | grep /data/influxdb-wal\r\n[root@host data]#\r\n```\r\n\r\nOur config:\r\n```\r\n[root@host telegraf]# grep wal /etc/influxdb/influxdb.conf\r\n  wal-dir = \"/data/influxdb-wal\"\r\n  max-wal-size = 10485760000\r\n  wal-flush-interval = \"30s\"\r\n  wal-partition-flush-delay = \"2s\"\r\n  wal-logging-enabled = true\r\n  wal-compaction-threshold = 0.5\r\n  wal-max-series-size = 1048576\r\n  wal-flush-cold-interval = \"5s\"\r\n  wal-flush-memory-size-threshold = 5242880\r\n  wal-max-memory-size-threshold = 10485760000\r\n  wal-partition-size-threshold = 1000000000\r\n  wal-ready-series-size = 512000\r\n```"},"comment":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/comments/150628947","html_url":"https://github.com/influxdb/influxdb/issues/4498#issuecomment-150628947","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/4498","id":150628947,"user":{"login":"daviesalex","id":1106249,"avatar_url":"https://avatars.githubusercontent.com/u/1106249?v=3","gravatar_id":"","url":"https://api.github.com/users/daviesalex","html_url":"https://github.com/daviesalex","followers_url":"https://api.github.com/users/daviesalex/followers","following_url":"https://api.github.com/users/daviesalex/following{/other_user}","gists_url":"https://api.github.com/users/daviesalex/gists{/gist_id}","starred_url":"https://api.github.com/users/daviesalex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/daviesalex/subscriptions","organizations_url":"https://api.github.com/users/daviesalex/orgs","repos_url":"https://api.github.com/users/daviesalex/repos","events_url":"https://api.github.com/users/daviesalex/events{/privacy}","received_events_url":"https://api.github.com/users/daviesalex/received_events","type":"User","site_admin":false},"created_at":"2015-10-23T16:41:40Z","updated_at":"2015-10-23T16:41:40Z","body":"We can confirm that #4388 is fixed in 0.9.5-nightly-827c513; our files are now rolling over at about 1.3GB.\r\n\r\nWe are still seeing these errors although far less frequently:\r\n```\r\n[tcp] 2015/10/23 11:37:43 process write shard error: write shard 4: engine: WAL backed up flushing to index, hit max memory\r\n```\r\n\r\nDespite now using PCIe flash for everything (IO service times are ~2ms max, %util never over 3%), but that is a separate issue - we will open a new issue.\r\n\r\nLets leave this issue to track the agreed change (make WAL directory configurable for tsm1).\r\n\r\nThanks! "}}