{"action":"created","issue":{"url":"https://api.github.com/repos/graphite-project/carbon/issues/371","labels_url":"https://api.github.com/repos/graphite-project/carbon/issues/371/labels{/name}","comments_url":"https://api.github.com/repos/graphite-project/carbon/issues/371/comments","events_url":"https://api.github.com/repos/graphite-project/carbon/issues/371/events","html_url":"https://github.com/graphite-project/carbon/issues/371","id":55629303,"number":371,"title":"carbon.agents.cache.size always increasing in 0.9.13pre1","user":{"login":"spencert","id":947938,"avatar_url":"https://avatars.githubusercontent.com/u/947938?v=3","gravatar_id":"","url":"https://api.github.com/users/spencert","html_url":"https://github.com/spencert","followers_url":"https://api.github.com/users/spencert/followers","following_url":"https://api.github.com/users/spencert/following{/other_user}","gists_url":"https://api.github.com/users/spencert/gists{/gist_id}","starred_url":"https://api.github.com/users/spencert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spencert/subscriptions","organizations_url":"https://api.github.com/users/spencert/orgs","repos_url":"https://api.github.com/users/spencert/repos","events_url":"https://api.github.com/users/spencert/events{/privacy}","received_events_url":"https://api.github.com/users/spencert/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/graphite-project/carbon/milestones/3","labels_url":"https://api.github.com/repos/graphite-project/carbon/milestones/3/labels","id":753094,"number":3,"title":"0.9.13","description":"","creator":{"login":"obfuscurity","id":494338,"avatar_url":"https://avatars.githubusercontent.com/u/494338?v=3","gravatar_id":"","url":"https://api.github.com/users/obfuscurity","html_url":"https://github.com/obfuscurity","followers_url":"https://api.github.com/users/obfuscurity/followers","following_url":"https://api.github.com/users/obfuscurity/following{/other_user}","gists_url":"https://api.github.com/users/obfuscurity/gists{/gist_id}","starred_url":"https://api.github.com/users/obfuscurity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/obfuscurity/subscriptions","organizations_url":"https://api.github.com/users/obfuscurity/orgs","repos_url":"https://api.github.com/users/obfuscurity/repos","events_url":"https://api.github.com/users/obfuscurity/events{/privacy}","received_events_url":"https://api.github.com/users/obfuscurity/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":24,"state":"open","created_at":"2014-08-14T05:06:20Z","updated_at":"2015-01-27T15:32:17Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2015-01-27T15:09:32Z","updated_at":"2015-01-27T15:36:12Z","closed_at":null,"body":"I am running the 0.9.13pre1 (specifically 36749eedce6be908109d5133bc173721d9662733) and I am noticing that the carbon.agents.*.cache.size is always increasing. \r\n\r\nMy setup is as follows. 2 physical servers, both with 32GB RAM and 10x15K SAS disks in a RAID10. For the metric load I am sending the servers disks keep up just fine.\r\n\r\nAll metrics come into server 1 which replicates metrics to itself (0.9.12 carbon) and server 2 (0.9.13pre1 carbon) over plain-text using carbon-c-relay. The carbon.conf for the [cache] section are exactly identical and I am running two caches on each physical server. As you can see I limit the MAX_UPDATES_PER_SECOND = 500 for each cache which makes sense on this hardware. However it is clear that 0.9.13pre1 cache.size always increases whereas 0.9.12 does not.\r\n\r\nI have attached two images which are the exact same metric targets just different hosts.\r\n\r\n![0 9 12-carbon](https://cloud.githubusercontent.com/assets/947938/5920539/9276de3c-a5fa-11e4-8ebb-43df8fbf3b18.png)\r\n![0 9 13pre1-carbon](https://cloud.githubusercontent.com/assets/947938/5920702/c47feb5c-a5fb-11e4-90bf-3ec0a444ac90.png)\r\n"},"comment":{"url":"https://api.github.com/repos/graphite-project/carbon/issues/comments/71667715","html_url":"https://github.com/graphite-project/carbon/issues/371#issuecomment-71667715","issue_url":"https://api.github.com/repos/graphite-project/carbon/issues/371","id":71667715,"user":{"login":"spencert","id":947938,"avatar_url":"https://avatars.githubusercontent.com/u/947938?v=3","gravatar_id":"","url":"https://api.github.com/users/spencert","html_url":"https://github.com/spencert","followers_url":"https://api.github.com/users/spencert/followers","following_url":"https://api.github.com/users/spencert/following{/other_user}","gists_url":"https://api.github.com/users/spencert/gists{/gist_id}","starred_url":"https://api.github.com/users/spencert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spencert/subscriptions","organizations_url":"https://api.github.com/users/spencert/orgs","repos_url":"https://api.github.com/users/spencert/repos","events_url":"https://api.github.com/users/spencert/events{/privacy}","received_events_url":"https://api.github.com/users/spencert/received_events","type":"User","site_admin":false},"created_at":"2015-01-27T15:36:12Z","updated_at":"2015-01-27T15:36:12Z","body":"It appears when I change the cache.size metric to be a nonNegativeDerivative it looks much better.  However, I am not sure this is accurate either...\r\n\r\n**0.9.13pre1**\r\nhttp://db5234.oak1.omniture.com/render/?width=586&height=308&_salt=1422370542.946&from=-14days&target=alias%28sumSeries%28carbon.agents.*.cache.queues%29%2C%22cache.queues%22%29&target=alias%28secondYAxis%28nonNegativeDerivative%28sumSeries%28carbon.agents.*.cache.size%29%29%29%2C%22cache.size%22%29&target=alias%28sumSeries%28carbon.agents.*.metricsReceived%29%2C%22Metrics%20Received%22%29&target=alias%28sumSeries%28carbon.agents.*.updateOperations%29%2C%22Update%20Operations%22%29&title=0.9.13pre1%20Carbon%20with%20nonNegativeDerivative%20applied%20for%20cache.size\r\n\r\n![0 9 13pre1-carbon-nonnegativederivative](https://cloud.githubusercontent.com/assets/947938/5921160/2c105574-a5ff-11e4-8f30-95627ab516f1.png)\r\n"}}