{"action":"opened","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3343","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3343/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3343/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3343/events","html_url":"https://github.com/zfsonlinux/zfs/issues/3343","id":70806255,"number":3343,"title":"ZFS Receive Panic and Pool Corruption","user":{"login":"talbottmd","id":7190845,"avatar_url":"https://avatars.githubusercontent.com/u/7190845?v=3","gravatar_id":"","url":"https://api.github.com/users/talbottmd","html_url":"https://github.com/talbottmd","followers_url":"https://api.github.com/users/talbottmd/followers","following_url":"https://api.github.com/users/talbottmd/following{/other_user}","gists_url":"https://api.github.com/users/talbottmd/gists{/gist_id}","starred_url":"https://api.github.com/users/talbottmd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talbottmd/subscriptions","organizations_url":"https://api.github.com/users/talbottmd/orgs","repos_url":"https://api.github.com/users/talbottmd/repos","events_url":"https://api.github.com/users/talbottmd/events{/privacy}","received_events_url":"https://api.github.com/users/talbottmd/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-04-24T22:06:09Z","updated_at":"2015-04-24T22:06:09Z","closed_at":null,"body":"I have a source system with CentOS 7 with zfs/spl 0.6.3-1 that I have not yet updated to 0.6.4. The replication backups haven't been working for me some quite time. Re: https://github.com/zfsonlinux/zfs/issues/2866\r\n\r\nOn the destination system, I've put a fresh install of CentOS 7 and zfs/spl 0.6.4. Destroyed the pool, wiped out the partition tables on all the disks, recreated the pool and started receiving again. Still getting a very reproducible problem, albeit a different crash now.\r\n\r\nBoth systems are Fujitsu PRIMERGY RX300s with 128G+ ECC RAM, 90 x 4TB SAS drives, and don't use dedup on any volumes. lz4 compression, Posix ACLs and extended attributes (sa) are all used.\r\n\r\nSending a filesystem from a remote server to this newly set up system eventually crashes. Seems to die around the same spot (around 1.1TB of 1.26TB on one particular filesystem) every time (and then around 80T of 120T on a different filesystem).\r\n\r\nIt'll reboot and not automatically import the pool. I can import the pool once using:\r\n>[root@store3 ~]# zpool import datastore\r\ncannot import 'datastore': I/O error\r\n\tRecovery is possible, but will result in some data loss.\r\n\tReturning the pool to its state as of Fri 24 Apr 2015 01:44:57 PM PDT\r\n\tshould correct the problem.  Approximately 1 seconds of data\r\n\tmust be discarded, irreversibly.  Recovery can be attempted\r\n\tby executing 'zpool import -F datastore'.  A scrub of the pool\r\n\tis strongly recommended after recovery.\r\n[root@store3 ~]# zpool import -F datastore\r\n\r\nThen I scrub the pool. Once the scrub is done, I roll it back a couple snapshots:\r\n>zfs rollback -fR datastore/Administration@zfs-auto-snap_weekly-2015-04-10-1002\r\n\r\nThen send an incremental over from the source to pick up where it left off. It'll crash very quickly, reboot, then the pool won't import again and reports:\r\n\r\n>[root@store3 ~]# zpool import datastore\r\ncannot import 'datastore': I/O error\r\nDestroy and re-create the pool from\r\na backup source.\r\n\r\nScary that receiving a zfs filesystem would somehow corrupt the entire pool! Thank goodness at least the sending side doesn't die.\r\n\r\nThe source pool is scrubbed every 2 weeks so I wouldn't think there'd be corruption there. Any suggestions are much appreciated.\r\n\r\nHere's a the relevant dmesg bits. I get the same call trace every time:\r\n\r\n>[  552.238938] BUG: unable to handle kernel NULL pointer dereference at 0000000000000060\r\n[  552.238992] IP: [<ffffffffa03a5630>] dsl_deadlist_insert+0x170/0x3a0 [zfs]\r\n[  552.239057] PGD 0 \r\n[  552.239071] Oops: 0000 [#1] SMP \r\n[  552.239092] Modules linked in: team_mode_loadbalance team dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sb_edac edac_core i2c_i801 mei_me lpc_ich mfd_core mei ses enclosure shpchp wmi ipmi_si ipmi_msghandler acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd sunrpc xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_common zfs(POF) zunicode(POF) zavl(POF) zcommon(POF) znvpair(POF) spl(OF) zlib_deflate mgag200 syscopyarea sysfillrect sysimgblt drm_kms_helper ahci ttm libahci ixgbe drm libata igb mdio i2c_algo_bit ptp i2c_core pps_core megaraid_sas dca mpt2sas(OF) raid_class scsi_transport_sas\r\n[  552.239490] CPU: 9 PID: 15759 Comm: txg_sync Tainted: PF          O--------------   3.10.0-229.1.2.el7.x86_64 #1\r\n[  552.239531] Hardware name: FUJITSU PRIMERGY RX300 S8/D2939-B1, BIOS V4.6.5.4 R1.14.0 for D2939-B1x 10/13/2014\r\n[  552.239571] task: ffff880fb38d0b60 ti: ffff880fb391c000 task.ti: ffff880fb391c000\r\n[  552.239603] RIP: 0010:[<ffffffffa03a5630>]  [<ffffffffa03a5630>] dsl_deadlist_insert+0x170/0x3a0 [zfs]\r\n[  552.239660] RSP: 0018:ffff880fb391f908  EFLAGS: 00010246\r\n[  552.239683] RAX: 0000000000000000 RBX: ffff881ff023cc38 RCX: 0000000000000001\r\n[  552.239713] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff882026269000\r\n[  552.239743] RBP: ffff880fb391f9d0 R08: 0000000000000202 R09: 00000001801a0019\r\n[  552.239774] R10: ffffffffa0061a07 R11: 0000000000000000 R12: ffff881fe62f7980\r\n[  552.239803] R13: ffff881fff863680 R14: 0000000000000000 R15: ffff881ff023ccb0\r\n[  552.239833] FS:  0000000000000000(0000) GS:ffff88207fc60000(0000) knlGS:0000000000000000\r\n[  552.239867] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033\r\n[  552.239891] CR2: 0000000000000060 CR3: 000000000190a000 CR4: 00000000001407e0\r\n[  552.239921] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000\r\n[  552.239951] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400\r\n[  552.239980] Stack:\r\n[  552.239992]  ffff8820260bac00 ffffffff816081b2 ffff881fe8159430 ffff880fb391f990\r\n[  552.240034]  ffffffffa0380b33 0000000000000000 000000000007955b ffff881fe3f6ad08\r\n[  552.240074]  ffffffffa037c913 ffff881fe203ee40 ffff881fe3f6ace0 ffff881fe8159488\r\n[  552.240112] Call Trace:\r\n[  552.240134]  [<ffffffff816081b2>] ? mutex_lock+0x12/0x2f\r\n[  552.240173]  [<ffffffffa0380b33>] ? dbuf_dirty+0x733/0xa40 [zfs]\r\n[  552.240211]  [<ffffffffa037c913>] ? dbuf_destroy+0x1d3/0x1e0 [zfs]\r\n[  552.240251]  [<ffffffffa03818d3>] ? dmu_buf_will_dirty+0xd3/0xf0 [zfs]\r\n[  552.240295]  [<ffffffffa039dfa8>] dsl_dataset_block_kill+0x218/0x460 [zfs]\r\n[  552.240348]  [<ffffffffa040b0ff>] ? zfs_space_delta_cb+0xcf/0x1a0 [zfs]\r\n[  552.240392]  [<ffffffffa039c0af>] free_blocks+0x11f/0x240 [zfs]\r\n[  552.240433]  [<ffffffffa039d16b>] dnode_sync+0x3fb/0x9d0 [zfs]\r\n[  552.240471]  [<ffffffffa0381e18>] ? dbuf_sync_list+0x58/0x80 [zfs]\r\n[  552.240499]  [<ffffffff816081b2>] ? mutex_lock+0x12/0x2f\r\n[  552.240537]  [<ffffffffa038c9bc>] dmu_objset_sync_dnodes+0x9c/0xc0 [zfs]\r\n[  552.240579]  [<ffffffffa038cbaf>] dmu_objset_sync+0x1cf/0x320 [zfs]\r\n[  552.240620]  [<ffffffffa038afb0>] ? redundant_metadata_changed_cb+0x20/0x20 [zfs]\r\n[  552.241640]  [<ffffffffa038cd00>] ? dmu_objset_sync+0x320/0x320 [zfs]\r\n[  552.242649]  [<ffffffffa039e561>] dsl_dataset_sync+0x41/0x50 [zfs]\r\n[  552.243660]  [<ffffffffa03accb3>] dsl_pool_sync+0xa3/0x420 [zfs]\r\n[  552.244678]  [<ffffffffa03c6d70>] spa_sync+0x480/0xbc0 [zfs]\r\n[  552.245654]  [<ffffffff8109836b>] ? autoremove_wake_function+0x2b/0x40\r\n[  552.246617]  [<ffffffff810a2484>] ? __wake_up+0x44/0x50\r\n[  552.247586]  [<ffffffffa03d8cee>] txg_sync_thread+0x37e/0x610 [zfs]\r\n[  552.248565]  [<ffffffffa03d8970>] ? txg_fini+0x2a0/0x2a0 [zfs]\r\n[  552.249511]  [<ffffffffa0061cc1>] thread_generic_wrapper+0x71/0x80 [spl]\r\n[  552.250432]  [<ffffffffa0061c50>] ? __thread_exit+0x20/0x20 [spl]\r\n[  552.251322]  [<ffffffff8109738f>] kthread+0xcf/0xe0\r\n[  552.252184]  [<ffffffff810972c0>] ? kthread_create_on_node+0x140/0x140\r\n[  552.253031]  [<ffffffff8161497c>] ret_from_fork+0x7c/0xb0\r\n[  552.253846]  [<ffffffff810972c0>] ? kthread_create_on_node+0x140/0x140\r\n[  552.254640] Code: 4c 89 f7 48 89 85 60 ff ff ff e8 1c 8b cd ff 31 d2 48 85 c0 0f 84 c1 01 00 00 4c 89 f7 48 89 c6 e8 d6 89 cd ff 49 89 c6 48 8b 3b <4d> 8b 7e 60 e8 97 7c fe ff 4c 3b b8 40 01 00 00 0f 84 ea 00 00 \r\n[  552.256346] RIP  [<ffffffffa03a5630>] dsl_deadlist_insert+0x170/0x3a0 [zfs]\r\n[  552.257159]  RSP <ffff880fb391f908>\r\n[  552.257933] CR2: 0000000000000060\r\n\r\nAny suggestions appreciated."}}