{"action":"opened","issue":{"url":"https://api.github.com/repos/facebook/flashcache/issues/212","labels_url":"https://api.github.com/repos/facebook/flashcache/issues/212/labels{/name}","comments_url":"https://api.github.com/repos/facebook/flashcache/issues/212/comments","events_url":"https://api.github.com/repos/facebook/flashcache/issues/212/events","html_url":"https://github.com/facebook/flashcache/issues/212","id":108342089,"number":212,"title":"Possible locking between multiple flashcache devices on one host","user":{"login":"byo","id":1365626,"avatar_url":"https://avatars.githubusercontent.com/u/1365626?v=3","gravatar_id":"","url":"https://api.github.com/users/byo","html_url":"https://github.com/byo","followers_url":"https://api.github.com/users/byo/followers","following_url":"https://api.github.com/users/byo/following{/other_user}","gists_url":"https://api.github.com/users/byo/gists{/gist_id}","starred_url":"https://api.github.com/users/byo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/byo/subscriptions","organizations_url":"https://api.github.com/users/byo/orgs","repos_url":"https://api.github.com/users/byo/repos","events_url":"https://api.github.com/users/byo/events{/privacy}","received_events_url":"https://api.github.com/users/byo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-09-25T14:57:45Z","updated_at":"2015-09-25T14:57:45Z","closed_at":null,"body":"Our test setup consists of 3 machines, each with 1 NVME and 12 HDD drives.\r\nNVME is split into 12 parts, each one serving as caching layer for associated HDD drive.\r\nMachines are used in ceph cluster, each NVME used as independently.\r\n\r\nNon-default sysctl variables on all devices are:\r\n```skip_seq_thresh_kb = 256```\r\n```reclaim_policy = 1```\r\n\r\nWe've observed periodical stalls on flashcache devices  during tests (basically no IO at all) lasting from few seconds to few minutes. Most of test io ops were crossing the sequential threshold. During such stall periods:\r\n- nr_queued value on all flashcache devices is large (>10k)\r\n- only one flashcache device is able to lower the nr_queued at a time - it's underlying HDD shows 100% util in iotop, all other HDD devices show some marginal values or no IO at all\r\n- if fallow_delay is set to 0,  perf top shows ```flashcache_deq_pending``` at the top, the hot spot is inside it's internal for loop (near ```if (node->index == index) {```)\r\n- if fallow_delay is set to 900, perf top shows either ```flachcache_clean_set``` or ```_raw_spin_lock_irq``` (called from flachcache_clean_set), hot spot is in one of flachcache_clean_set's for loops (near ```if (!(cacheblk->cache_state & DIRTY_FALLOW_2))```)\r\n\r\nFrom what I read in the code, I understand that flashcache uses global kernel thread pools for it's job. Is it possible all cleaning jobs are executed on one core only? That could explain why only one HDD can clean itself at the same time.\r\n\r\nWhen ```skip_seq_thresh_kb = 0``` and ```fallow_delay = 0```, cluster can handle it's load without issues. If the dirty percent is near 99%, queues start filling up but they're cleaned up simultaneously and multiple HDDs are at 100% util.\r\n\r\nSysctl dump of a sample drive (configuration causing issues):\r\n\r\n```\r\ncache_all = 1\r\nclean_on_read_miss = 0\r\nclean_on_write_miss = 0\r\ndirty_thresh_pct = 20\r\ndo_pid_expiry = 0\r\ndo_sync = 0\r\nfallow_clean_speed = 2\r\nfallow_delay = 900\r\nfast_remove = 1\r\nio_latency_hist = 0\r\nlru_hot_pct = 75\r\nlru_promote_thresh = 2\r\nmax_clean_ios_set = 2\r\nmax_clean_ios_total = 4\r\nmax_pids = 100\r\nnew_style_write_merge = 0\r\npid_expiry_secs = 60\r\nreclaim_policy = 1\r\nskip_seq_thresh_kb = 256\r\nstop_sync = 0\r\nzero_stats = 0\r\n```\r\n"}}