{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/spl/issues/500","labels_url":"https://api.github.com/repos/zfsonlinux/spl/issues/500/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/spl/issues/500/comments","events_url":"https://api.github.com/repos/zfsonlinux/spl/issues/500/events","html_url":"https://github.com/zfsonlinux/spl/issues/500","id":116971490,"number":500,"title":"locking in taskq_member() hurts performance on large-scale systems","user":{"login":"dweeezil","id":1432005,"avatar_url":"https://avatars.githubusercontent.com/u/1432005?v=3","gravatar_id":"","url":"https://api.github.com/users/dweeezil","html_url":"https://github.com/dweeezil","followers_url":"https://api.github.com/users/dweeezil/followers","following_url":"https://api.github.com/users/dweeezil/following{/other_user}","gists_url":"https://api.github.com/users/dweeezil/gists{/gist_id}","starred_url":"https://api.github.com/users/dweeezil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dweeezil/subscriptions","organizations_url":"https://api.github.com/users/dweeezil/orgs","repos_url":"https://api.github.com/users/dweeezil/repos","events_url":"https://api.github.com/users/dweeezil/events{/privacy}","received_events_url":"https://api.github.com/users/dweeezil/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-11-15T04:11:15Z","updated_at":"2015-12-02T20:46:36Z","closed_at":null,"body":"On large-scale systems with lots of CPU cores, the locking in ```taskq_member()``` becomes a significant contention point for zios dispatched to the single ```z_wr_iss``` taskq for heavy (high-frequency) IO workloads.  Under illumos, this operation is a single comparison operation whereas under ZoL, it involves taking a lock, iterating through a list and then releasing the lock.\r\n\r\nThis mainly affects the ```z_wr_iss``` taskq because it's the main zio taskq with a high volume of dispatches but for which only a single taskq is allocated.  Unfortunately, the obvious workaround of something like this:\r\n```\r\ndiff --git a/module/zfs/spa.c b/module/zfs/spa.c\r\nindex 2e23a34..9480c2b 100644\r\n--- a/module/zfs/spa.c\r\n+++ b/module/zfs/spa.c\r\n@@ -128,7 +128,7 @@ const zio_taskq_info_t zio_taskqs[ZIO_TYPES][ZIO_TASKQ_TYPES] = {\r\n        /* ISSUE        ISSUE_HIGH      INTR            INTR_HIGH */\r\n        { ZTI_ONE,      ZTI_NULL,       ZTI_ONE,        ZTI_NULL }, /* NULL */\r\n        { ZTI_N(8),     ZTI_NULL,       ZTI_P(12, 8),   ZTI_NULL }, /* READ */\r\n-       { ZTI_BATCH,    ZTI_N(5),       ZTI_P(12, 8),   ZTI_N(5) }, /* WRITE */\r\n+       { ZTI_P(4, 10), ZTI_N(5),       ZTI_P(12, 8),   ZTI_N(5) }, /* WRITE */\r\n        { ZTI_P(12, 8), ZTI_NULL,       ZTI_ONE,        ZTI_NULL }, /* FREE */\r\n        { ZTI_ONE,      ZTI_NULL,       ZTI_ONE,        ZTI_NULL }, /* CLAIM */\r\n        { ZTI_ONE,      ZTI_NULL,       ZTI_ONE,        ZTI_NULL }, /* IOCTL */\r\n```\r\nseems to expose other bottlenecks during high-throughput benchmarking and actually reduces performance.  I've not yet discovered why this happens but the above patch definitely reduces the impact of the locking in ```taskq_member()```."},"comment":{"url":"https://api.github.com/repos/zfsonlinux/spl/issues/comments/161429264","html_url":"https://github.com/zfsonlinux/spl/issues/500#issuecomment-161429264","issue_url":"https://api.github.com/repos/zfsonlinux/spl/issues/500","id":161429264,"user":{"login":"dweeezil","id":1432005,"avatar_url":"https://avatars.githubusercontent.com/u/1432005?v=3","gravatar_id":"","url":"https://api.github.com/users/dweeezil","html_url":"https://github.com/dweeezil","followers_url":"https://api.github.com/users/dweeezil/followers","following_url":"https://api.github.com/users/dweeezil/following{/other_user}","gists_url":"https://api.github.com/users/dweeezil/gists{/gist_id}","starred_url":"https://api.github.com/users/dweeezil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dweeezil/subscriptions","organizations_url":"https://api.github.com/users/dweeezil/orgs","repos_url":"https://api.github.com/users/dweeezil/repos","events_url":"https://api.github.com/users/dweeezil/events{/privacy}","received_events_url":"https://api.github.com/users/dweeezil/received_events","type":"User","site_admin":false},"created_at":"2015-12-02T20:46:36Z","updated_at":"2015-12-02T20:46:36Z","body":"@behlendorf I'll definitely include your new patch, which looks perfectly reasonable to me at first glance."}}