{"action":"created","issue":{"url":"https://api.github.com/repos/graphite-project/graphite-web/issues/1314","labels_url":"https://api.github.com/repos/graphite-project/graphite-web/issues/1314/labels{/name}","comments_url":"https://api.github.com/repos/graphite-project/graphite-web/issues/1314/comments","events_url":"https://api.github.com/repos/graphite-project/graphite-web/issues/1314/events","html_url":"https://github.com/graphite-project/graphite-web/issues/1314","id":104749125,"number":1314,"title":"Graphite  performance issue","user":{"login":"ravibhooshan","id":11981903,"avatar_url":"https://avatars.githubusercontent.com/u/11981903?v=3","gravatar_id":"","url":"https://api.github.com/users/ravibhooshan","html_url":"https://github.com/ravibhooshan","followers_url":"https://api.github.com/users/ravibhooshan/followers","following_url":"https://api.github.com/users/ravibhooshan/following{/other_user}","gists_url":"https://api.github.com/users/ravibhooshan/gists{/gist_id}","starred_url":"https://api.github.com/users/ravibhooshan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ravibhooshan/subscriptions","organizations_url":"https://api.github.com/users/ravibhooshan/orgs","repos_url":"https://api.github.com/users/ravibhooshan/repos","events_url":"https://api.github.com/users/ravibhooshan/events{/privacy}","received_events_url":"https://api.github.com/users/ravibhooshan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-09-03T17:52:03Z","updated_at":"2015-09-08T15:31:01Z","closed_at":null,"body":"Hi All,\r\nI need some expert suggestion on improving graphite performance.\r\nI am running below architecture:\r\nOne relay behind F5 VIP\r\n4 carbon servers\r\nRelay is using consistent hashing and replication factor -4 \r\nAll are running on physical server on centos with 6 CPUs (4 cores each), 64GB RAM and 800GB SSD.\r\nNo of carbon creates - 75\r\nrest all settings as default.\r\nTill 500-600k metrics per minutes i was not having any issue.\r\nNow we are hitting 1.5 Million metric / minute.\r\n\r\nOn each release day we are adding 300-400k metric and during release  (in 30 minutes) we used to push around 300-400k metrics and that is where problem starts.\r\nData starts queuing up and see 80% data getting dropped at relay. It takes around 2-3 days to normalize.\r\n\r\nI thought it because of carbon-cache is not completing metric in  time so planned to run multiple caches and crested 3 caches on 2 of carbon servers. But that is also not solving the issue.\r\n\r\nOther issue i facing is that in multi-cache i am running child relay on carbon server using rules.file. But relays rules is not directing data on the basis of regex.\r\n\r\nPlease let me know how to improve data flow from relay to carbon-cache. Please let me knwo if you have question."},"comment":{"url":"https://api.github.com/repos/graphite-project/graphite-web/issues/comments/138601169","html_url":"https://github.com/graphite-project/graphite-web/issues/1314#issuecomment-138601169","issue_url":"https://api.github.com/repos/graphite-project/graphite-web/issues/1314","id":138601169,"user":{"login":"ashokrajar","id":1329679,"avatar_url":"https://avatars.githubusercontent.com/u/1329679?v=3","gravatar_id":"","url":"https://api.github.com/users/ashokrajar","html_url":"https://github.com/ashokrajar","followers_url":"https://api.github.com/users/ashokrajar/followers","following_url":"https://api.github.com/users/ashokrajar/following{/other_user}","gists_url":"https://api.github.com/users/ashokrajar/gists{/gist_id}","starred_url":"https://api.github.com/users/ashokrajar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ashokrajar/subscriptions","organizations_url":"https://api.github.com/users/ashokrajar/orgs","repos_url":"https://api.github.com/users/ashokrajar/repos","events_url":"https://api.github.com/users/ashokrajar/events{/privacy}","received_events_url":"https://api.github.com/users/ashokrajar/received_events","type":"User","site_admin":false},"created_at":"2015-09-08T15:31:01Z","updated_at":"2015-09-08T15:31:01Z","body":"You will not hit the IO/Mem unless you are running as many carbon-cache servers per cpu core and they are stressed by millions & millions of metrics.\r\n\r\nTake a look at this blog which explains how to do clustering with graphite. I would strongly propose such setup  for you. I also have a similar setup in production but I have replaced the relay component with carbon-c-relay for performance and data loss issues. When carbon-relay/carbon-cache is under stress and when it hit the cpu limit(GIL) it stops accepting new metrics.\r\n\r\nThis is not a issue. It's more of a knowledge sharing & best practices in setting up a graphite cluster. So please close this issue and move this discussion to the mailing-list/IRC."}}