{"action":"reopened","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3443","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3443/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3443/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3443/events","html_url":"https://github.com/zfsonlinux/zfs/issues/3443","id":81067383,"number":3443,"title":"PANIC at dbuf.c:105:dbuf_dest()","user":{"login":"nedbass","id":320381,"avatar_url":"https://avatars.githubusercontent.com/u/320381?v=3","gravatar_id":"","url":"https://api.github.com/users/nedbass","html_url":"https://github.com/nedbass","followers_url":"https://api.github.com/users/nedbass/followers","following_url":"https://api.github.com/users/nedbass/following{/other_user}","gists_url":"https://api.github.com/users/nedbass/gists{/gist_id}","starred_url":"https://api.github.com/users/nedbass/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nedbass/subscriptions","organizations_url":"https://api.github.com/users/nedbass/orgs","repos_url":"https://api.github.com/users/nedbass/repos","events_url":"https://api.github.com/users/nedbass/events{/privacy}","received_events_url":"https://api.github.com/users/nedbass/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Bug%20-%20Major","name":"Bug - Major","color":"e11d21"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/zfsonlinux/zfs/milestones/11","html_url":"https://github.com/zfsonlinux/zfs/milestones/0.7.0","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/milestones/11/labels","id":7013,"number":11,"title":"0.7.0","description":"ZFS on Linux v0.7.0 - Memory management\r\n\r\n","creator":{"login":"behlendorf","id":148917,"avatar_url":"https://avatars.githubusercontent.com/u/148917?v=3","gravatar_id":"","url":"https://api.github.com/users/behlendorf","html_url":"https://github.com/behlendorf","followers_url":"https://api.github.com/users/behlendorf/followers","following_url":"https://api.github.com/users/behlendorf/following{/other_user}","gists_url":"https://api.github.com/users/behlendorf/gists{/gist_id}","starred_url":"https://api.github.com/users/behlendorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/behlendorf/subscriptions","organizations_url":"https://api.github.com/users/behlendorf/orgs","repos_url":"https://api.github.com/users/behlendorf/repos","events_url":"https://api.github.com/users/behlendorf/events{/privacy}","received_events_url":"https://api.github.com/users/behlendorf/received_events","type":"User","site_admin":false},"open_issues":147,"closed_issues":115,"state":"open","created_at":"2011-05-03T17:38:46Z","updated_at":"2015-09-25T20:00:02Z","due_on":null,"closed_at":null},"comments":7,"created_at":"2015-05-26T17:50:39Z","updated_at":"2015-09-25T20:00:02Z","closed_at":null,"body":"Reproduced on master 4 commits past 0.6.4 (7fad629). Also 77 commits past 0.6.4 (65037d9). Process running `unlink()` destroys a dbuf while the `txg_sync` thread still holds `db->db_mtx`.\r\n\r\nReproducer was using 64 concurrent copies of xattrtest https://github.com/nedbass/xattrtest/tree/x.  The ASSERT hits reliably during the unlink  phase. I used this test script:\r\n\r\n```\r\ndoit()\r\n{\r\n        local test=$1\r\n        local dir=$2\r\n        for ((i=0;i<64;i++)) ; do\r\n                mkdir -p $dir/$i\r\n                ( ./xattrtest -o $test -f 1024 -R -s 512 -p $dir/$i -v -d > $dir/$i/xattrtest.log ) &\r\n        done\r\n        wait\r\n        awk '/\\/second$/ {print  $4}' $dir/*/xattrtest.log | awk '{sum += $1} END {print sum}'\r\n}\r\n\r\nd=`mktemp -d /mnt/fish/XXXXXXXX`\r\necho -n \" create: \"\r\n( doit 1 $d )\r\necho -n \" setxattr: \"\r\n( doit 2 $d )\r\necho -n \" getxattr: \"\r\n( doit 3 $d )\r\necho -n \" unlink: \"\r\n( doit 4 $d )\r\necho\r\n```\r\n\r\n```\r\n2015-05-26 10:21:44 ZFS: Loaded module v0.6.4-4_g7fad629 (DEBUG mode), ZFS pool version 5000, ZFS filesystem version 5\r\n2015-05-26 10:22:25 SPL: using hostid 0xa8c08c40\r\n2015-05-26 10:26:45 VERIFY3(((*(volatile typeof((&db->db_mtx)->m_owner) *)&((&db->db_mtx)->m_owner))) == ((void *)0)) failed (ffff88202691e040 == (null))\r\n2015-05-26 10:26:45 PANIC at dbuf.c:105:dbuf_dest()\r\nShowing stack for process 60942\r\n```\r\n\r\n```\r\ncrash> bt 60942\r\nPID: 60942  TASK: ffff881015300040  CPU: 28  COMMAND: \"xattrtest\"\r\n #0 [ffff88101364d7e8] schedule at ffffffff8152d740\r\n #1 [ffff88101364d8c0] spl_panic at ffffffffa08fd23d [spl]\r\n #2 [ffff88101364da60] dbuf_dest at ffffffffa0a6a3c8 [zfs]\r\n #3 [ffff88101364da80] spl_kmem_cache_free at ffffffffa08f9e06 [spl]\r\n #4 [ffff88101364dac0] dbuf_destroy at ffffffffa0a6a7b9 [zfs]\r\n #5 [ffff88101364db00] dbuf_evict at ffffffffa0a6d62a [zfs]\r\n #6 [ffff88101364db20] dnode_evict_bonus at ffffffffa0a9d2a7 [zfs]\r\n #7 [ffff88101364db50] dbuf_rele_and_unlock at ffffffffa0a6d0bb [zfs]\r\n #8 [ffff88101364dc00] dbuf_rele at ffffffffa0a6d2d5 [zfs]\r\n #9 [ffff88101364dc20] dmu_buf_rele at ffffffffa0a6e57e [zfs]\r\n#10 [ffff88101364dc30] sa_handle_destroy at ffffffffa0aca4ec [zfs]\r\n#11 [ffff88101364dc50] zfs_znode_dmu_fini at ffffffffa0b4a91f [zfs]\r\n#12 [ffff88101364dc90] zfs_znode_delete at ffffffffa0b4ad3e [zfs]\r\n#13 [ffff88101364dce0] zfs_rmnode at ffffffffa0b21358 [zfs]\r\n#14 [ffff88101364dd20] zfs_zinactive at ffffffffa0b4abb4 [zfs]\r\n#15 [ffff88101364dd60] zfs_inactive at ffffffffa0b4161f [zfs]\r\n#16 [ffff88101364ddb0] zpl_clear_inode at ffffffffa0b66f82 [zfs]\r\n#17 [ffff88101364ddd0] clear_inode at ffffffff811ad8e2\r\n#18 [ffff88101364ddf0] zpl_inode_delete at ffffffffa0b66c80 [zfs]\r\n#19 [ffff88101364de10] generic_delete_inode at ffffffff811adfce\r\n#20 [ffff88101364de40] zpl_drop_inode at ffffffffa0b66c9e [zfs]\r\n#21 [ffff88101364de50] iput at ffffffff811acf62\r\n#22 [ffff88101364de70] do_unlinkat at ffffffff811a1b39\r\n#23 [ffff88101364df70] sys_unlink at ffffffff811a1c06\r\n#24 [ffff88101364df80] system_call_fastpath at ffffffff8100b0b2\r\n    RIP: 00002aaaaada9657  RSP: 00007fffffffd988  RFLAGS: 00010246\r\n    RAX: 0000000000000057  RBX: ffffffff8100b0b2  RCX: 0000000000000000\r\n    RDX: 0000000000000000  RSI: 00000000004032fc  RDI: 0000000000616010\r\n    RBP: 00007fffffffe2c0   R8: 0000000000000000   R9: 0000000000000000\r\n    R10: 0000000000000000  R11: 0000000000000246  R12: ffffffff811a1c06\r\n    R13: ffff88101364df78  R14: 0000000000000000  R15: 0000000000000000\r\n    ORIG_RAX: 0000000000000057  CS: 0033  SS: 002b\r\nll_fastpath at ffffffff8100b0b2\r\n    RIP: 00002aaaaada9657  RSP: 00007fffffffd988  RFLAGS: 00010246\r\n    RAX: 0000000000000057  RBX: ffffffff8100b0b2  RCX: 0000000000000000\r\n    RDX: 0000000000000000  RSI: 00000000004032fc  RDI: 0000000000616010\r\n    RBP: 00007fffffffe2c0   R8: 0000000000000000   R9: 0000000000000000\r\n    R10: 0000000000000000  R11: 0000000000000246  R12: ffffffff811a1c06\r\n    R13: ffff88101364df78  R14: 0000000000000000  R15: 0000000000000000\r\n    ORIG_RAX: 0000000000000057  CS: 0033  SS: 002b\r\n```\r\n\r\nHere we see `txg_sync` is the mutex owner:\r\n\r\n```\r\ncrash> task ffff88202691e040 | head -1\r\nPID: 59618  TASK: ffff88202691e040  CPU: 25  COMMAND: \"txg_sync\"\r\n```\r\n\r\n```\r\n[root@zwicky-lcw-mds1:xattrtest]# uname -a\r\nLinux zwicky-lcw-mds1 2.6.32-504.12.2.2chaos.ch5.3.x86_64 #1 SMP Fri Mar 20 13:20:55 PDT 2015 x86_64 x86_64 x86_64 GNU/Linux\r\n```"}}