{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3616","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3616/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3616/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3616/events","html_url":"https://github.com/zfsonlinux/zfs/issues/3616","id":96078490,"number":3616,"title":"ZFS IO can stall for a second or more under memory pressure","user":{"login":"siebenmann","id":865382,"avatar_url":"https://avatars.githubusercontent.com/u/865382?v=3","gravatar_id":"","url":"https://api.github.com/users/siebenmann","html_url":"https://github.com/siebenmann","followers_url":"https://api.github.com/users/siebenmann/followers","following_url":"https://api.github.com/users/siebenmann/following{/other_user}","gists_url":"https://api.github.com/users/siebenmann/gists{/gist_id}","starred_url":"https://api.github.com/users/siebenmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/siebenmann/subscriptions","organizations_url":"https://api.github.com/users/siebenmann/orgs","repos_url":"https://api.github.com/users/siebenmann/repos","events_url":"https://api.github.com/users/siebenmann/events{/privacy}","received_events_url":"https://api.github.com/users/siebenmann/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-07-20T14:38:49Z","updated_at":"2015-07-20T17:19:33Z","closed_at":null,"body":"In arc_get_data_buf, if arc_is_overflowing() is true we signal the reclaim thread and wait for arc_reclaim_waiters_cv. On my machine with git tip ZFS on Linux, I'm periodically seeing multiple processes stalling in this wait for a second or more; this has the effect of drastically slowing down their IO.\r\n\r\nThis CV is primarily signalled from two places. In arc_evict_state_impl(), it is signalled on !arc_is_overflowing(); in arc_adapt_thread() (which winds up calling arc_evict_state_impl()) it is signalled on 'arc_size <= arc_c || arc_evicted = 0'. However, these two conditions are not quite the same because arc_is_overflowing() adds an overflow block to arc_c. So if arc_evict_state_impl() evicts some space but not the full overflow amount, it will not signal the CV but we'll wind up with arc_evicted > 0, arc_adapt_thread() may not signal the CV either because arc_size is now a bit over arc_c, and arc_adapt_thread() will then wind up sleeping for a second before it tries again and maybe signals the CV this time around. In the mean time, arc_get_data_buf() callers are waiting.\r\n\r\nThis issue was introduced in ca0bf58d65f77e944b9905571df9a2eae647aeca, a port of the upstream 'Illumos 5497 - lock contention on arcs_mtx'. The issue is not present in the upstream Illumos code because there the entire cv_timedwait_sig() section is also in the 'arc_size <= arc_c || arc_evicted == 0' conditional, so arc_adapt_thread() always signals if it's going to wait (and otherwise it immediately tries to shrink again).\r\n\r\n(I don't know if the best fix is to make the code work like the Illumos code or to change the condition in arc_adapt_thread() to also be '!arc_is_overflowing()' or something like that. The Illumos code clearly pushes ARC shrinking more, since it will keep trying to evict things until nothing is evicted.)"},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/122957529","html_url":"https://github.com/zfsonlinux/zfs/issues/3616#issuecomment-122957529","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3616","id":122957529,"user":{"login":"dweeezil","id":1432005,"avatar_url":"https://avatars.githubusercontent.com/u/1432005?v=3","gravatar_id":"","url":"https://api.github.com/users/dweeezil","html_url":"https://github.com/dweeezil","followers_url":"https://api.github.com/users/dweeezil/followers","following_url":"https://api.github.com/users/dweeezil/following{/other_user}","gists_url":"https://api.github.com/users/dweeezil/gists{/gist_id}","starred_url":"https://api.github.com/users/dweeezil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dweeezil/subscriptions","organizations_url":"https://api.github.com/users/dweeezil/orgs","repos_url":"https://api.github.com/users/dweeezil/repos","events_url":"https://api.github.com/users/dweeezil/events{/privacy}","received_events_url":"https://api.github.com/users/dweeezil/received_events","type":"User","site_admin":false},"created_at":"2015-07-20T17:19:33Z","updated_at":"2015-07-20T17:19:33Z","body":"I'm going to run some metadata-intensive test in tight memory environments to make sure the arc size doesn't blow up."}}