{"action":"created","issue":{"url":"https://api.github.com/repos/openzfsonosx/zfs/issues/255","labels_url":"https://api.github.com/repos/openzfsonosx/zfs/issues/255/labels{/name}","comments_url":"https://api.github.com/repos/openzfsonosx/zfs/issues/255/comments","events_url":"https://api.github.com/repos/openzfsonosx/zfs/issues/255/events","html_url":"https://github.com/openzfsonosx/zfs/issues/255","id":50104070,"number":255,"title":"unlinked_drain: panic in zap_cursor_retrieve","user":{"login":"lundman","id":637168,"avatar_url":"https://avatars.githubusercontent.com/u/637168?v=3","gravatar_id":"","url":"https://api.github.com/users/lundman","html_url":"https://github.com/lundman","followers_url":"https://api.github.com/users/lundman/followers","following_url":"https://api.github.com/users/lundman/following{/other_user}","gists_url":"https://api.github.com/users/lundman/gists{/gist_id}","starred_url":"https://api.github.com/users/lundman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lundman/subscriptions","organizations_url":"https://api.github.com/users/lundman/orgs","repos_url":"https://api.github.com/users/lundman/repos","events_url":"https://api.github.com/users/lundman/events{/privacy}","received_events_url":"https://api.github.com/users/lundman/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/openzfsonosx/zfs/labels/bug","name":"bug","color":"fc2929"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2014-11-26T00:40:13Z","updated_at":"2015-01-07T00:01:24Z","closed_at":"2014-11-28T15:41:49Z","body":"\r\nO3X has been running without unlinked_drain since 1.3.0, and we recently turned it on. We have seen pools with number of unlinked-list entries from 67k to 1.3M. The issue is, sometimes this list contains entries that are bad.\r\n\r\nFor the ones I have looked at, they are XATTR dirs, where we try to iterate the entries with zap to delete them, but something goes wrong.\r\n\r\nPanic reports look like\r\n\r\n```\r\n0xffffff81ec333640 : 0xffffff800343a443 mach_kernel : _return_from_trap + 0xe3\r\n0xffffff81ec333660 : 0xffffff7f83e9519c net.lundman.zfs : _zap_leaf_lookup_closest + 0x8c\r\n0xffffff81ec333790 : 0xffffff7f83e927f4 net.lundman.zfs : _fzap_cursor_retrieve + 0x174\r\n0xffffff81ec333810 : 0xffffff7f83e9a754 net.lundman.zfs : _zap_cursor_retrieve + 0x164\r\n0xffffff81ec3338b0 : 0xffffff7f83ea9853 net.lundman.zfs : _zfs_purgedir + 0x73\r\n0xffffff81ec333a80 : 0xffffff7f83ea941c net.lundman.zfs : _zfs_rmnode + 0x7c\r\n0xffffff81ec333ae0 : 0xffffff7f83ee8e16 net.lundman.zfs : _zfs_zinactive + 0x136\r\n0xffffff81ec333b10 : 0xffffff7f83edbb2c net.lundman.zfs : _zfs_vnop_reclaim + 0x1ec\r\n0xffffff81ec333b90 : 0xffffff800354b870 mach_kernel : _vfs_addtrigger + 0xf70\r\n0xffffff81ec333c10 : 0xffffff800354b35a mach_kernel : _vfs_addtrigger + 0xa5a\r\n0xffffff81ec333c60 : 0xffffff80035458f4 mach_kernel : _vflush + 0x274\r\n0xffffff81ec333cd0 : 0xffffff800355089a mach_kernel : _dounmount + 0x37a\r\n0xffffff81ec333d70 : 0xffffff80035502bb mach_kernel : _unmount + 0x17b\r\n```\r\n\r\n```\r\n<lundman> zap_get_leaf_byblk (in zfs) (zap.c:529)\r\n<lundman> zap_deref_leaf (in zfs) (zap.c:585)\r\n<lundman> fzap_cursor_retrieve (in zfs) (zap.c:1181)\r\n<lundman> zap_cursor_retrieve (in zfs) (zap_micro.c:1278)\r\n<lundman> zfs_purgedir (in zfs) (zfs_dir.c:594)\r\n<lundman> zfs_rmnode (in zfs) (zfs_dir.c:667)\r\n```\r\n\r\n```\r\n0xffffff80be89b000 : 0xffffff7f8a36211c net.lundman.zfs : _zap_leaf_lookup_closest + 0x8c\r\n0xffffff80be89b130 : 0xffffff7f8a35f774 net.lundman.zfs : _fzap_cursor_retrieve + 0x174\r\n0xffffff80be89b1b0 : 0xffffff7f8a3676d4 net.lundman.zfs : _zap_cursor_retrieve + 0x164\r\n0xffffff80be89b250 : 0xffffff7f8a3767d3 net.lundman.zfs : _zfs_purgedir + 0x73\r\n0xffffff80be89b420 : 0xffffff7f8a37639c net.lundman.zfs : _zfs_rmnode + 0x7c\r\n0xffffff80be89b480 : 0xffffff7f8a3b5db6 net.lundman.zfs : _zfs_zinactive + 0x136\r\n0xffffff80be89b4b0 : 0xffffff7f8a3a8acc net.lundman.zfs : _zfs_vnop_reclaim + 0x\r\n```\r\n\r\n```\r\npanic(cpu 3 caller 0xffffff7f97482512): \"rwlock 0xffffff80b0a4d080 not initialised\\n\"@spl-rwlock.c:74\r\nBacktrace (CPU 3), Frame : Return Address\r\n0xffffff80b68bac60 : 0xffffff8015b3a811 mach_kernel : _panic + 0xd1\r\n0xffffff80b68bace0 : 0xffffff7f97482512 net.lundman.spl : _rw_enter + 0x32\r\n0xffffff80b68bad10 : 0xffffff7f9755f843 net.lundman.zfs : _zap_get_leaf_byblk + 0xb3\r\n0xffffff80b68bad70 : 0xffffff7f9755d3cc net.lundman.zfs : _zap_deref_leaf + 0xbc\r\n0xffffff80b68bade0 : 0xffffff7f9755ef3a net.lundman.zfs : _fzap_cursor_retrieve + 0x11a\r\n0xffffff80b68bae60 : 0xffffff7f97566ef4 net.lundman.zfs : _zap_cursor_retrieve + 0x164\r\n0xffffff80b68baf00 : 0xffffff7f97575fb3 net.lundman.zfs : _zfs_purgedir + 0x73\r\n0xffffff80b68bb0d0 : 0xffffff7f97575bbc net.lundman.zfs : _zfs_rmnode + 0x7c\r\n0xffffff80b68bb130 : 0xffffff7f975b4fe8 net.lundman.zfs : _zfs_zinactive + 0xe8\r\n0xffffff80b68bb160 : 0xffffff7f975a7d78 net.lundman.zfs : _zfs_vnop_reclaim + 0x258\r\n```\r\n\r\n\r\nThe annoying thing is that `zfs_unlinked_drain()` is called at mount time, and it just iterates the list calling `zget()`. \r\n\r\nSometime later, when the kernel decides to call `vnop_reclaim()` on them do we crash. It can be never on a completely unused system, hours on mostly idle, and minutes on a busy pool. And not related to the task that was performed at the time.\r\n\r\nWe could go back to not processing unlinked-drain list, but there is a fair bit of space being lost, more some on Apple where XATTRs are used more frequently.\r\n\r\nI added a kstat switch `zfs_vnop_skip_unlinked_drain` so to skip the unlinked drain, and use the pool without (a delayed) panic.\r\n\r\nhttps://github.com/openzfsonosx/zfs/commit/4ea7019efd4d9941794edacec1876561cbb95b73\r\n\r\nsysctl -w kstat.zfs.darwin.tunable.skip_unlinked_drain=1\r\n\r\n\r\n \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/openzfsonosx/zfs/issues/comments/68957377","html_url":"https://github.com/openzfsonosx/zfs/issues/255#issuecomment-68957377","issue_url":"https://api.github.com/repos/openzfsonosx/zfs/issues/255","id":68957377,"user":{"login":"lundman","id":637168,"avatar_url":"https://avatars.githubusercontent.com/u/637168?v=3","gravatar_id":"","url":"https://api.github.com/users/lundman","html_url":"https://github.com/lundman","followers_url":"https://api.github.com/users/lundman/followers","following_url":"https://api.github.com/users/lundman/following{/other_user}","gists_url":"https://api.github.com/users/lundman/gists{/gist_id}","starred_url":"https://api.github.com/users/lundman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lundman/subscriptions","organizations_url":"https://api.github.com/users/lundman/orgs","repos_url":"https://api.github.com/users/lundman/repos","events_url":"https://api.github.com/users/lundman/events{/privacy}","received_events_url":"https://api.github.com/users/lundman/received_events","type":"User","site_admin":false},"created_at":"2015-01-07T00:01:24Z","updated_at":"2015-01-07T00:01:24Z","body":"Worth noting the `rw_lock` hackery isn't needed and was removed. Only needed to test against ZAP_MAGIC and return EIO to detect it. The patch could be cleaned up if wanted.\r\n"}}