{"action":"created","issue":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/4119","labels_url":"https://api.github.com/repos/influxdb/influxdb/issues/4119/labels{/name}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/4119/comments","events_url":"https://api.github.com/repos/influxdb/influxdb/issues/4119/events","html_url":"https://github.com/influxdb/influxdb/issues/4119","id":106680149,"number":4119,"title":"0.9.4 WAL death spiral","user":{"login":"kev009","id":90042,"avatar_url":"https://avatars.githubusercontent.com/u/90042?v=3","gravatar_id":"","url":"https://api.github.com/users/kev009","html_url":"https://github.com/kev009","followers_url":"https://api.github.com/users/kev009/followers","following_url":"https://api.github.com/users/kev009/following{/other_user}","gists_url":"https://api.github.com/users/kev009/gists{/gist_id}","starred_url":"https://api.github.com/users/kev009/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kev009/subscriptions","organizations_url":"https://api.github.com/users/kev009/orgs","repos_url":"https://api.github.com/users/kev009/repos","events_url":"https://api.github.com/users/kev009/events{/privacy}","received_events_url":"https://api.github.com/users/kev009/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2015-09-16T01:45:07Z","updated_at":"2015-09-17T20:08:51Z","closed_at":null,"body":"Hi,\r\n\r\nTrying to see if Influx can replace a TSDB cluster.  Right now, just one decently spec'd hw system and no clustering.\r\n\r\nThe wal starts growing, then RAM usage goes up until swapping and it's unusable.  ZFS and the disks are bored.  It looks like one 'shard' is being overused but I haven't looked into how WAL and boltdb work/dont't work together yet.\r\n\r\nI'm wondering if we could perhaps skip things like compression and compaction on filesystems like ZFS?\r\n\r\n```\r\nroot@influx-1:/data/influxdb/data/llnw-phx7-move/default # ls -lh /data/influxdb/data/llnw-phx7-move/default/16 \r\n-rw-r--r--  1 influxd  influxd    12G Sep 15 18:19 /data/influxdb/data/llnw-phx7-move/default/16\r\n```\r\n\r\n```\r\n[opentsdb] 2015/09/15 18:38:30 failed to write point batch to database \"llnw-phx7-move\": write failed: engine: write points: write throughput too high. backoff and retry\r\n[write] 2015/09/15 18:38:30 write failed for shard 16 on node 1: engine: write points: write throughput too high. backoff and retry\r\n[opentsdb] 2015/09/15 18:38:30 failed to write point batch to database \"llnw-phx7-move\": write failed: engine: write points: write throughput too high. backoff and retry\r\n```\r\n\r\n```\r\n[data]\r\n  dir = \"{{ datadir }}\"\r\n\r\n  # The following WAL settings are for the b1 storage engine used in 0.9.2. They won't\r\n  # apply to any new shards created after upgrading to a version > 0.9.3.\r\n  max-wal-size = 104857600 # Maximum size the WAL can reach before a flush. Defaults to 100MB.\r\n  wal-flush-interval = \"10m\" # Maximum time data can sit in WAL before a flush.\r\n  wal-partition-flush-delay = \"2s\" # The delay time between each WAL partition being flushed.\r\n\r\n  # These are the WAL settings for the storage engine >= 0.9.3\r\n  wal-dir = \"{{ waldir }}\"\r\n  wal-enable-logging = true\r\n\r\n  # When a series in the WAL in-memory cache reaches this size in bytes it is marked as ready to\r\n  # flush to the index\r\n  # wal-ready-series-size = 25600\r\n\r\n  # Flush and compact a partition once this ratio of series are over the ready size\r\n  # wal-compaction-threshold = 0.6\r\n\r\n  # Force a flush and compaction if any series in a partition gets above this size in bytes\r\n  # wal-max-series-size = 2097152\r\n\r\n  # Force a flush of all series and full compaction if there have been no writes in this\r\n  # amount of time. This is useful for ensuring that shards that are cold for writes don't\r\n  # keep a bunch of data cached in memory and in the WAL.\r\n  # wal-flush-cold-interval = \"10m\"\r\n\r\n  # Force a partition to flush its largest series if it reaches this approximate size in\r\n  # bytes. Remember there are 5 partitions so you'll need at least 5x this amount of memory.\r\n  # The more memory you have, the bigger this can be.\r\n  # wal-partition-size-threshold = 20971520\r\n\r\n[opentsdb]\r\n  enabled = true\r\n  # bind-address = \"\"\r\n  database = \"llnw-phx7-move\"\r\n  # retention-policy = \"\"\r\n```\r\n```"},"comment":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/comments/141212261","html_url":"https://github.com/influxdb/influxdb/issues/4119#issuecomment-141212261","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/4119","id":141212261,"user":{"login":"christianchristensen","id":65640,"avatar_url":"https://avatars.githubusercontent.com/u/65640?v=3","gravatar_id":"","url":"https://api.github.com/users/christianchristensen","html_url":"https://github.com/christianchristensen","followers_url":"https://api.github.com/users/christianchristensen/followers","following_url":"https://api.github.com/users/christianchristensen/following{/other_user}","gists_url":"https://api.github.com/users/christianchristensen/gists{/gist_id}","starred_url":"https://api.github.com/users/christianchristensen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/christianchristensen/subscriptions","organizations_url":"https://api.github.com/users/christianchristensen/orgs","repos_url":"https://api.github.com/users/christianchristensen/repos","events_url":"https://api.github.com/users/christianchristensen/events{/privacy}","received_events_url":"https://api.github.com/users/christianchristensen/received_events","type":"User","site_admin":false},"created_at":"2015-09-17T20:08:51Z","updated_at":"2015-09-17T20:08:51Z","body":"Just for good measure: `X-Influxdb-Version: 0.9` and `SHOW DIAGNOSTICS` returns \"0.9\""}}