{"action":"created","issue":{"url":"https://api.github.com/repos/ceph/ceph/issues/4792","labels_url":"https://api.github.com/repos/ceph/ceph/issues/4792/labels{/name}","comments_url":"https://api.github.com/repos/ceph/ceph/issues/4792/comments","events_url":"https://api.github.com/repos/ceph/ceph/issues/4792/events","html_url":"https://github.com/ceph/ceph/pull/4792","id":81927096,"number":4792,"title":"Wip writeback throttling for cache tiering","user":{"login":"dragonylffly","id":4452507,"avatar_url":"https://avatars.githubusercontent.com/u/4452507?v=3","gravatar_id":"","url":"https://api.github.com/users/dragonylffly","html_url":"https://github.com/dragonylffly","followers_url":"https://api.github.com/users/dragonylffly/followers","following_url":"https://api.github.com/users/dragonylffly/following{/other_user}","gists_url":"https://api.github.com/users/dragonylffly/gists{/gist_id}","starred_url":"https://api.github.com/users/dragonylffly/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dragonylffly/subscriptions","organizations_url":"https://api.github.com/users/dragonylffly/orgs","repos_url":"https://api.github.com/users/dragonylffly/repos","events_url":"https://api.github.com/users/dragonylffly/events{/privacy}","received_events_url":"https://api.github.com/users/dragonylffly/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ceph/ceph/labels/core","name":"core","color":"0052cc"},{"url":"https://api.github.com/repos/ceph/ceph/labels/feature","name":"feature","color":"009800"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-05-28T13:32:25Z","updated_at":"2015-05-28T16:49:10Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ceph/ceph/pulls/4792","html_url":"https://github.com/ceph/ceph/pull/4792","diff_url":"https://github.com/ceph/ceph/pull/4792.diff","patch_url":"https://github.com/ceph/ceph/pull/4792.patch"},"body":"This patch is to do write back throttling for cache tiering, which is similar to what the Linux kernel does for page cache write back. The motivation and original idea are proposed by Nick Fisk, detailed in his email as below. In our implementation, we introduce a paramter 'cache_target_dirty_high_ratio'  (default 0.6) as the high speed threshold, while leave the\r\n'cache_target_dirty_ratio' (default 0.4) to represent the low speed  threshold, we control the flush speed by limiting the parallelism of flushing. The maximum parallelism under low speed is half of the parallelism under high speed. If there is at least one PG such that the dirty ratio beyond the high threshold, full speed mode is entered; If there is no PG such that dirty ratio beyond the low threshold, idle mode is entered; In other cases, slow speed mode is entered.\r\n\r\n-------- Original Message --------\r\nSubject: \tCeph Tiering Idea\r\nDate: \tFri, 22 May 2015 16:07:46 +0100\r\nFrom: \tNick Fisk <nick@fisk.me.uk>\r\nTo: \tliwang@ubuntukylin.com\r\n\r\nHi,\r\n\r\nI’ve just seen your post to the Ceph Dev Mailing list regarding adding \r\ntemperature based eviction to the cache eviction logic. I think this is \r\na much needed enhancement and can’t wait to test it out once it hits the \r\nnext release.\r\n\r\nI have been testing Ceph Cache Tiering for a number of months now and \r\nanother enhancement which I think would greatly enhance the performance \r\nwould be high and low thresholds for flushing and eviction. I have tried \r\nlooking through the Ceph source, but with my limited programming skills \r\nI was unable to make any progress and so thought I would share my idea \r\nwith you and get your thoughts.\r\n\r\nCurrently as soon as you exceed the flush/eviction threshold, Ceph \r\nstarts aggressively flushing to the base tier which impacts performance. \r\nFor long running write operations this is probably unavoidable, however \r\nmost workloads are normally quite bursty and my idea of having high and \r\nlow thresholds would hopefully improve performance where the writes come \r\nin bursts.\r\n\r\nWhen the cache tier approaches the low threshold, Ceph would start \r\nflushing/evicting with a low priority, so performance is not affected. \r\nIf the high threshold is reached, Ceph will flush more aggressively, \r\nsimilar to the current behaviour. Hopefully during the quiet periods \r\nin-between bursts of writes, the cache would slowly be reduced down to \r\nthe low threshold meaning it is ready for the next burst.\r\n\r\nFor example:-\r\n\r\n1TB Cache Tier\r\n\r\nLow Dirty=0.4\r\n\r\nHigh Dirty=0.6\r\n\r\nCache tier would contain 400GB of dirty data at idle, as dirty data \r\nrises above 400GB, Ceph would flush with a low priority or throttled \r\nMB/s rate.\r\n\r\nIf Cache tier raises above 600GB, Ceph will aggressively flush to keep \r\ndirty data below 60%\r\n\r\nThe above should give you 200GB capacity of bursty writes before \r\nperformance becomes impacted\r\n\r\nDoes this make sense?\r\n\r\nMany Thanks,\r\n\r\nNick\r\n\r\n\r\n\r\n\r\n \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/ceph/ceph/issues/comments/106482306","html_url":"https://github.com/ceph/ceph/pull/4792#issuecomment-106482306","issue_url":"https://api.github.com/repos/ceph/ceph/issues/4792","id":106482306,"user":{"login":"liewegas","id":433031,"avatar_url":"https://avatars.githubusercontent.com/u/433031?v=3","gravatar_id":"","url":"https://api.github.com/users/liewegas","html_url":"https://github.com/liewegas","followers_url":"https://api.github.com/users/liewegas/followers","following_url":"https://api.github.com/users/liewegas/following{/other_user}","gists_url":"https://api.github.com/users/liewegas/gists{/gist_id}","starred_url":"https://api.github.com/users/liewegas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liewegas/subscriptions","organizations_url":"https://api.github.com/users/liewegas/orgs","repos_url":"https://api.github.com/users/liewegas/repos","events_url":"https://api.github.com/users/liewegas/events{/privacy}","received_events_url":"https://api.github.com/users/liewegas/received_events","type":"User","site_admin":false},"created_at":"2015-05-28T16:49:10Z","updated_at":"2015-05-28T16:49:10Z","body":"I kind of wish we could make this a smooth function instead of a step between high and low.  There is the effort calculations, for example.  But.. this has to happen across the whole OSD, so those effort values don't work very well.  So this is probably the best we can do with the current infrastructure.\r\n"}}