{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3555","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3555/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3555/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3555/events","html_url":"https://github.com/zfsonlinux/zfs/pull/3555","id":92709474,"number":3555,"title":"Illumos 5008 lock contention (rrw_exit) while running a read only load","user":{"login":"dweeezil","id":1432005,"avatar_url":"https://avatars.githubusercontent.com/u/1432005?v=3","gravatar_id":"","url":"https://api.github.com/users/dweeezil","html_url":"https://github.com/dweeezil","followers_url":"https://api.github.com/users/dweeezil/followers","following_url":"https://api.github.com/users/dweeezil/following{/other_user}","gists_url":"https://api.github.com/users/dweeezil/gists{/gist_id}","starred_url":"https://api.github.com/users/dweeezil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dweeezil/subscriptions","organizations_url":"https://api.github.com/users/dweeezil/orgs","repos_url":"https://api.github.com/users/dweeezil/repos","events_url":"https://api.github.com/users/dweeezil/events{/privacy}","received_events_url":"https://api.github.com/users/dweeezil/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-07-02T19:01:06Z","updated_at":"2015-07-02T19:04:02Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/zfsonlinux/zfs/pulls/3555","html_url":"https://github.com/zfsonlinux/zfs/pull/3555","diff_url":"https://github.com/zfsonlinux/zfs/pull/3555.diff","patch_url":"https://github.com/zfsonlinux/zfs/pull/3555.patch"},"body":"5008 lock contention (rrw_exit) while running a read only load\r\nReviewed by: Matthew Ahrens <matthew.ahrens@delphix.com>\r\nReviewed by: George Wilson <george.wilson@delphix.com>\r\nReviewed by: Alex Reece <alex.reece@delphix.com>\r\nReviewed by: Christopher Siden <christopher.siden@delphix.com>\r\nReviewed by: Richard Yao <ryao@gentoo.org>\r\nReviewed by: Saso Kiselkov <skiselkov.ml@gmail.com>\r\nApproved by: Garrett D'Amore <garrett@damore.org>\r\nPorted by: Tim Chase <tim@chase2k.com>\r\n\r\nPorting notes:\r\n\r\nThis patch ported prefectly cleanly to ZoL.  During testing 100% cached\r\nsmall-block reads, extreme contention was noticed on rrl->rr_lock from\r\nrrw_exit() due to the frequent entering and leaving ZPL.  Illumos picked\r\nup this patch from FreeBSD and it also helps under Linux.\r\n\r\nOn a 1-minute 4K cached read test with 10 fio processes pinned to a single\r\nsocket on a 4-socket (10 thread per socket) NUMA system, contentions on\r\nrrl->rr_lock were reduced from 508799 to 43085."},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/118131716","html_url":"https://github.com/zfsonlinux/zfs/pull/3555#issuecomment-118131716","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3555","id":118131716,"user":{"login":"dweeezil","id":1432005,"avatar_url":"https://avatars.githubusercontent.com/u/1432005?v=3","gravatar_id":"","url":"https://api.github.com/users/dweeezil","html_url":"https://github.com/dweeezil","followers_url":"https://api.github.com/users/dweeezil/followers","following_url":"https://api.github.com/users/dweeezil/following{/other_user}","gists_url":"https://api.github.com/users/dweeezil/gists{/gist_id}","starred_url":"https://api.github.com/users/dweeezil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dweeezil/subscriptions","organizations_url":"https://api.github.com/users/dweeezil/orgs","repos_url":"https://api.github.com/users/dweeezil/repos","events_url":"https://api.github.com/users/dweeezil/events{/privacy}","received_events_url":"https://api.github.com/users/dweeezil/received_events","type":"User","site_admin":false},"created_at":"2015-07-02T19:04:02Z","updated_at":"2015-07-02T19:04:02Z","body":"As mentioned in the commit comment, I was noticing large amounts of contention due to frequent rrw_enter()/rrw_exit() calls when lots of system calls were being made.  Further investigation led me to this patch from Illumos via FreeBSD.  Unfortunately, although it's a huge help for lock contention, it doesn't make a dramatic impact on performance in my testing.  It definitely does help, however, and is for example, easily noticable when running ```htop``` in the lower amount of kernel time being consumed by each process."}}