{"action":"created","issue":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/4289","labels_url":"https://api.github.com/repos/influxdb/influxdb/issues/4289/labels{/name}","comments_url":"https://api.github.com/repos/influxdb/influxdb/issues/4289/comments","events_url":"https://api.github.com/repos/influxdb/influxdb/issues/4289/events","html_url":"https://github.com/influxdb/influxdb/issues/4289","id":109347656,"number":4289,"title":"Shards not getting created, throwing write failed errors (throughput too high)","user":{"login":"sebito91","id":3159454,"avatar_url":"https://avatars.githubusercontent.com/u/3159454?v=3","gravatar_id":"","url":"https://api.github.com/users/sebito91","html_url":"https://github.com/sebito91","followers_url":"https://api.github.com/users/sebito91/followers","following_url":"https://api.github.com/users/sebito91/following{/other_user}","gists_url":"https://api.github.com/users/sebito91/gists{/gist_id}","starred_url":"https://api.github.com/users/sebito91/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sebito91/subscriptions","organizations_url":"https://api.github.com/users/sebito91/orgs","repos_url":"https://api.github.com/users/sebito91/repos","events_url":"https://api.github.com/users/sebito91/events{/privacy}","received_events_url":"https://api.github.com/users/sebito91/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-10-01T17:38:25Z","updated_at":"2015-10-01T19:03:21Z","closed_at":null,"body":"We have an 0.9.4.1 cluster (stable release) of three nodes with a default retention policy as defined by telegraf (v0.1.9, also stable release), and it seems unable to keep up with the data load. In particular, we're seeing thousands of messages the error below, and no extra shard created:\r\n\r\n```[tcp] 2015/10/01 12:07:16 process write shard error: write shard 1: engine: write points: write throughput too high. backoff and retry```\r\n\r\nLeaving the metrics to collect overnight, we returned in the morning to see one single shard at >95GB on each node. We've combed through a lot of the posts, including tweaks like #3885 to no avail. Even turning off HH and drastically reducing the wal knobs end up seeing a lot of gaps in the data, if it's committed at all.\r\n\r\nReading up on #4086 and similar threads, I'm curious if we're bumping up against the locking mechanism during wal flush. It seems that once we trigger a flush, we block all incoming read/write requests and just never recover. Unfortunately our setup represents only a fraction of the nodes we need to collect from so currently the backend is just unable to keep up.\r\n\r\nKey Questions:\r\n\r\n1. Can we turn off the wal at all, buffering into memory/cache instead? We have incredibly powerful machines (it's not a hardware problem) that can handle this type of throughput provided we can get it at the disks...\r\n2. @parisholley, thanks! ~~Does shard-precreation actually work? Doesn't seem to generate more than 1 shard, ever?~~\r\n3. To improve throughput does it make sense to tweak the LB to be sticky until a node dies? Will replication occur async to each of the cluster nodes?\r\n4. When can we enable shards across more than 3 nodes, this is a massive limitation at this point and we can only scale up so far?\r\n5. If you want to comb logs, do you have drop box we can send them to? We're quite eager to get this working properly...\r\n\r\nCONFIG:\r\n* ~1200 machines reporting through haproxy (roundrobin), ~60Mbit/sec\r\n* 64bit RHEL 7.1\r\n* default telegraf collectors (v0.1.9 stable): cpu disk io mem net swap system\r\n* collect every 10s\r\n* three backend influxdb nodes (v0.9.4.1 stable, very beefy SSD-backed machines, 256GB RAM each)\r\n\r\n```\r\n### Welcome to the InfluxDB configuration file.\r\n\r\n# Once every 24 hours InfluxDB will report anonymous data to m.influxdb.com\r\n# The data includes raft id (random 8 bytes), os, arch, version, and metadata.\r\n# We don't track ip addresses of servers reporting. This is only used\r\n# to track the number of instances running and the versions, which\r\n# is very helpful for us.\r\n# Change this option to true to disable reporting.\r\nreporting-disabled = true\r\n\r\n###\r\n### [meta]\r\n###\r\n### Controls the parameters for the Raft consensus group that stores metadata\r\n### about the InfluxDB cluster.\r\n###\r\n\r\n[meta]\r\n  dir = \"/data/influxdb/meta\"\r\n  hostname = \"testmetricsl6\"\r\n  bind-address = \":8088\"\r\n  retention-autocreate = true\r\n  election-timeout = \"1s\"\r\n  heartbeat-timeout = \"1s\"\r\n  leader-lease-timeout = \"500ms\"\r\n  commit-timeout = \"50ms\"\r\n  cluster-tracing = true\r\n\r\n###\r\n### [data]\r\n###\r\n### Controls where the actual shard data for InfluxDB lives.\r\n###\r\n\r\n[data]\r\n  dir = \"/data/influxdb/metrics\"\r\n  max-wal-size = 10485760\r\n  wal-flush-interval = \"1m\"\r\n  wal-partition-flush-delay = \"2s\"\r\n  wal-dir = \"/data/influxdb/wal\"\r\n  wal-logging-enabled = true\r\n  wal-ready-series-size = 1024 \r\n  wal-compaction-threshold = 0.5\r\n  wal-max-series-size = 1048576\r\n  wal-flush-cold-interval = \"5s\"\r\n  wal-partition-size-threshold = 10485760 \r\n\r\n###\r\n### [cluster]\r\n###\r\n### Controls non-Raft cluster behavior, which generally includes how data is\r\n### shared across shards.\r\n###\r\n\r\n[cluster]\r\n  force-remote-mapping = false \r\n  write-timeout = \"2s\"\r\n  shard-writer-timeout = \"2s\"\r\n  shard-mapper-timeout = \"2s\"\r\n\r\n###\r\n### [retention]\r\n###\r\n### Controls the enforcement of retention policies for evicting old data.\r\n###\r\n\r\n[retention]\r\n  enabled = true\r\n  check-interval = \"10m0s\"\r\n\r\n[shard-precreation]\r\n  enabled = true\r\n  check-interval = \"30s\"\r\n  advance-period = \"60m0s\"\r\n\r\n###\r\n### [admin]\r\n###\r\n### Controls the availability of the built-in, web-based admin interface.\r\n###\r\n\r\n[admin]\r\n  enabled = true\r\n  bind-address = \":8083\"\r\n  https-enabled = false\r\n  https-certificate = \"/etc/ssl/influxdb.pem\"\r\n\r\n###\r\n### [http]\r\n###\r\n### Controls how the HTTP endpoints are configured. These are the primary\r\n### mechanism for getting data into and out of InfluxDB.\r\n###\r\n\r\n[http]\r\n  enabled = true\r\n  bind-address = \":8086\"\r\n  auth-enabled = false\r\n  log-enabled = true\r\n  write-tracing = false\r\n  pprof-enabled = false\r\n  https-enabled = false\r\n  https-certificate = \"/etc/ssl/influxdb.pem\"\r\n\r\n###\r\n### [[graphite]]\r\n###\r\n### Controls one or many listeners for Graphite data.\r\n###\r\n\r\n[[graphite]]\r\n  enabled = false\r\n  bind-address = \":2003\"\r\n  database = \"graphite\"\r\n  protocol = \"tcp\"\r\n  batch-size = 1000\r\n  batch-timeout = \"1s\"\r\n  tcp-conn-timeout = \"15m0s\"\r\n  consistency-level = \"one\"\r\n  separator = \".\"\r\n\r\n###\r\n### [collectd]\r\n###\r\n### Controls the listener for collectd data.\r\n###\r\n\r\n[collectd]\r\n  enabled = false\r\n  bind-address = \":25826\"\r\n  database = \"collectd\"\r\n  retention-policy = \"\"\r\n  batch-size = 5000\r\n  batch-pending = 5\r\n  batch-timeout = \"10s\"\r\n  typesdb = \"/usr/share/collectd/types.db\"\r\n\r\n###\r\n### [opentsdb]\r\n###\r\n### Controls the listener for OpenTSDB data.\r\n###\r\n\r\n[opentsdb]\r\n  enabled = false\r\n  bind-address = \":4242\"\r\n  database = \"opentsdb\"\r\n  retention-policy = \"\"\r\n  consistency-level = \"one\"\r\n  tls-enabled = false\r\n  certificate = \"/etc/ssl/influxdb.pem\"\r\n  batch-size = 1000\r\n  batch-pending = 5\r\n  batch-timeout = \"1s\"\r\n\r\n###\r\n### [monitor]\r\n###\r\n\r\n[monitor]\r\n  store-enabled = false \r\n  store-database = \"_internal\"\r\n  store-interval = \"10s\"\r\n\r\n###\r\n### [continuous_queries]\r\n###\r\n### Controls how continuous queries are run within InfluxDB.\r\n###\r\n\r\n[continuous_queries]\r\n  enabled = true\r\n  log-enabled = true\r\n  recompute-previous-n = 2\r\n  recompute-no-older-than = \"10m0s\"\r\n  compute-runs-per-interval = 10\r\n  compute-no-more-than = \"2m0s\"\r\n\r\n###\r\n### [hinted-handoff]\r\n###\r\n### Controls the hinted handoff feature, which allows nodes to temporarily\r\n### store queued data when one node of a cluster is down for a short period\r\n### of time.\r\n###\r\n\r\n[hinted-handoff]\r\n  enabled = false \r\n  dir = \"/data/influxdb/hh\"\r\n  max-size = 1073741824\r\n  max-age = \"168h0m0s\"\r\n  retry-rate-limit = 0\r\n  retry-interval = \"1s\"\r\n```"},"comment":{"url":"https://api.github.com/repos/influxdb/influxdb/issues/comments/144817980","html_url":"https://github.com/influxdb/influxdb/issues/4289#issuecomment-144817980","issue_url":"https://api.github.com/repos/influxdb/influxdb/issues/4289","id":144817980,"user":{"login":"sebito91","id":3159454,"avatar_url":"https://avatars.githubusercontent.com/u/3159454?v=3","gravatar_id":"","url":"https://api.github.com/users/sebito91","html_url":"https://github.com/sebito91","followers_url":"https://api.github.com/users/sebito91/followers","following_url":"https://api.github.com/users/sebito91/following{/other_user}","gists_url":"https://api.github.com/users/sebito91/gists{/gist_id}","starred_url":"https://api.github.com/users/sebito91/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sebito91/subscriptions","organizations_url":"https://api.github.com/users/sebito91/orgs","repos_url":"https://api.github.com/users/sebito91/repos","events_url":"https://api.github.com/users/sebito91/events{/privacy}","received_events_url":"https://api.github.com/users/sebito91/received_events","type":"User","site_admin":false},"created_at":"2015-10-01T19:03:21Z","updated_at":"2015-10-01T19:03:21Z","body":"So with @parisholley's guidance, we do see shards getting created on the hour (likely based on our config). That being said, the other 4 questions still remain and throughput issues are still present."}}