{"action":"closed","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/1761","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/1761/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/1761/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/1761/events","html_url":"https://github.com/zfsonlinux/zfs/issues/1761","id":20286576,"number":1761,"title":"Zfs incremental recv fails with dataset is busy error","user":{"login":"olw2005","id":3035339,"avatar_url":"https://avatars.githubusercontent.com/u/3035339?v=3","gravatar_id":"","url":"https://api.github.com/users/olw2005","html_url":"https://github.com/olw2005","followers_url":"https://api.github.com/users/olw2005/followers","following_url":"https://api.github.com/users/olw2005/following{/other_user}","gists_url":"https://api.github.com/users/olw2005/gists{/gist_id}","starred_url":"https://api.github.com/users/olw2005/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olw2005/subscriptions","organizations_url":"https://api.github.com/users/olw2005/orgs","repos_url":"https://api.github.com/users/olw2005/repos","events_url":"https://api.github.com/users/olw2005/events{/privacy}","received_events_url":"https://api.github.com/users/olw2005/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Bug%20-%20Minor","name":"Bug - Minor","color":"e11d21"},{"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Difficulty%20-%20Medium","name":"Difficulty - Medium","color":"207de5"},{"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/zvol","name":"zvol","color":"02e10c"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":24,"created_at":"2013-09-30T19:35:57Z","updated_at":"2015-04-14T05:04:18Z","closed_at":"2015-04-14T05:04:18Z","body":"Dear All,\r\nFirst let me say, keep up the great work!  Happy to see my tax dollars funding something useful.\r\n\r\nBrief problem description:\r\nWe have two servers setup to send rotating incremental snapshots of the local zvol to each other (zfs send -I).  In the main this works well, but about once a week the zfs recv fails with a 'dataset is busy' error.\r\n\r\nDmesg output is:\r\nBuffer I/O error on device zd16, logical block 0\r\nBuffer I/O error on device zd16, logical block 0\r\nBuffer I/O error on device zd16, logical block 0\r\n\r\nExpected dmesg output:\r\n zd16:\r\nGPT:Primary header thinks Alt. header is not at the end of the disk.\r\nGPT:34358689719 != 34359738367\r\nGPT:Alternate GPT header not at the end of the disk.\r\nGPT:34358689719 != 34359738367\r\nGPT: Use GNU Parted to correct GPT errors.\r\n p1\r\n(The zvol is used as the disk for a drbd resource.  Drbd uses the last bit of the disk to store its own metadata, presenting a slightly smaller disk to layers above it.  Thus the spurious GPT warning is normal / expected.)\r\n\r\nAfter this error, all subsequent zfs recv's on that particular zvol [pool/chicagorepl] with the same error.  Only way to clear the error is to reboot the server.  Meanwhile, the two other zvols [arch/archive and pool/detroit] on the server continue to work fine, creating / sending / destroying snapshots w/o issue.\r\n\r\nServer Configuration / Setup Info:\r\nHP DL380p Gen8, 192 GB RAM, four external disk chassis [48 disks], two ssd's setup as l2arc cache disks\r\nCentOS 6.4 with recent updates\r\nzfs & spl 0.6.2\r\n\r\n[root@dtc-san2 ~]# uname -a\r\nLinux dtc-san2.stc.ricplc.com 2.6.32-358.18.1.el6.x86_64 #1 SMP Wed Aug 28 17:19:38 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\n[root@dtc-san2 ~]# cat /etc/modprobe.d/zfs.conf\r\noptions zfs zfs_nocacheflush=1 zfs_arc_max=154618822656 zfs_arc_min=1073741824\r\n\r\n[root@dtc-san2 ~]# zpool status\r\n  pool: arch\r\n state: ONLINE\r\n  scan: none requested\r\nconfig:\r\n\r\n        NAME        STATE     READ WRITE CKSUM\r\n        arch        ONLINE       0     0     0\r\n          raidz2-0  ONLINE       0     0     0\r\n            sdan    ONLINE       0     0     0\r\n            sdao    ONLINE       0     0     0\r\n            sdap    ONLINE       0     0     0\r\n            sdaq    ONLINE       0     0     0\r\n            sdar    ONLINE       0     0     0\r\n            sdas    ONLINE       0     0     0\r\n            sdat    ONLINE       0     0     0\r\n            sdau    ONLINE       0     0     0\r\n            sdav    ONLINE       0     0     0\r\n            sdaw    ONLINE       0     0     0\r\n            sdax    ONLINE       0     0     0\r\n        spares\r\n          sday      AVAIL\r\n\r\nerrors: No known data errors\r\n\r\n  pool: pool\r\n state: ONLINE\r\n  scan: none requested\r\nconfig:\r\n\r\n        NAME        STATE     READ WRITE CKSUM\r\n        pool        ONLINE       0     0     0\r\n          raidz2-0  ONLINE       0     0     0\r\n            sdd     ONLINE       0     0     0\r\n            sde     ONLINE       0     0     0\r\n            sdf     ONLINE       0     0     0\r\n            sdg     ONLINE       0     0     0\r\n            sdh     ONLINE       0     0     0\r\n            sdi     ONLINE       0     0     0\r\n            sdj     ONLINE       0     0     0\r\n            sdk     ONLINE       0     0     0\r\n            sdl     ONLINE       0     0     0\r\n            sdm     ONLINE       0     0     0\r\n          raidz2-1  ONLINE       0     0     0\r\n            sdp     ONLINE       0     0     0\r\n            sdq     ONLINE       0     0     0\r\n            sdr     ONLINE       0     0     0\r\n            sds     ONLINE       0     0     0\r\n            sdt     ONLINE       0     0     0\r\n            sdu     ONLINE       0     0     0\r\n            sdv     ONLINE       0     0     0\r\n            sdw     ONLINE       0     0     0\r\n            sdx     ONLINE       0     0     0\r\n            sdy     ONLINE       0     0     0\r\n          raidz2-2  ONLINE       0     0     0\r\n            sdab    ONLINE       0     0     0\r\n            sdac    ONLINE       0     0     0\r\n            sdae    ONLINE       0     0     0\r\n            sdaf    ONLINE       0     0     0\r\n            sdag    ONLINE       0     0     0\r\n            sdah    ONLINE       0     0     0\r\n            sdai    ONLINE       0     0     0\r\n            sdaj    ONLINE       0     0     0\r\n            sdak    ONLINE       0     0     0\r\n        cache\r\n          sdb       ONLINE       0     0     0\r\n          sdc       ONLINE       0     0     0\r\n        spares\r\n          sdn       AVAIL\r\n          sdo       AVAIL\r\n          sdz       AVAIL\r\n          sdaa      AVAIL\r\n          sdal      AVAIL\r\n          sdam      AVAIL\r\n\r\nerrors: No known data errors\r\n\r\n[root@dtc-san2 ~]# zfs list\r\nNAME               USED  AVAIL  REFER  MOUNTPOINT\r\narch              19.8T  4.20T  65.8K  none\r\narch/archive      19.8T  4.20T  19.1T  -\r\npool              20.8T  40.6T  64.5K  none\r\npool/chicagorepl  5.98T  40.6T  4.77T  -\r\npool/detroit      14.6T  40.6T  13.8T  -\r\npool/vdistore      177G  40.6T   177G  -\r\n\r\nAbbreviated zpool history:\r\nHistory for 'arch':\r\n...\r\n2013-09-30.14:16:07 zfs snapshot arch/archive@hour-20130930141601\r\n2013-09-30.14:16:33 zfs send -I arch/archive@hour-20130930131601 arch/archive@hour-20130930141601\r\n2013-09-30.14:28:10 zfs destroy arch/archive@hour-20130930101601\r\n2013-09-30.15:16:07 zfs snapshot arch/archive@hour-20130930151601\r\n2013-09-30.15:16:31 zfs send -I arch/archive@hour-20130930141601 arch/archive@hour-20130930151601\r\n\r\nHistory for 'pool':\r\n...\r\n2013-09-30.12:05:11 zfs recv -F pool/chicagorepl\r\n2013-09-30.12:05:33 zfs destroy pool/chicagorepl@hour-20130930070101\r\n2013-09-30.12:14:13 zfs destroy pool/detroit@hour-20130930064601\r\n2013-09-30.12:14:20 zfs destroy pool/detroit@hour-20130930074601\r\n2013-09-30.12:42:06 zfs snapshot pool/detroit@4hour-20130930124201\r\n2013-09-30.12:42:32 zfs destroy pool/detroit@4hour-20130929124201\r\n2013-09-30.12:46:06 zfs snapshot pool/detroit@hour-20130930124601\r\n2013-09-30.12:46:49 zfs send -I pool/detroit@hour-20130930114601 pool/detroit@hour-20130930124601\r\n2013-09-30.12:54:15 zfs destroy pool/detroit@hour-20130930084601\r\n2013-09-30.13:20:01 zfs recv -F pool/chicagorepl\r\n2013-09-30.13:20:20 zfs destroy pool/chicagorepl@hour-20130930080101\r\n2013-09-30.13:46:07 zfs snapshot pool/detroit@hour-20130930134601\r\n2013-09-30.13:46:53 zfs send -I pool/detroit@hour-20130930124601 pool/detroit@hour-20130930134601\r\n2013-09-30.14:00:29 zfs destroy pool/detroit@hour-20130930094601\r\n2013-09-30.14:01:23 zfs destroy pool/chicagorepl@4hour-20130929125601\r\n---->  2013-09-30.14:02:40 zfs recv -F pool/chicagorepl\r\n2013-09-30.14:46:06 zfs snapshot pool/detroit@hour-20130930144601\r\n2013-09-30.14:46:47 zfs send -I pool/detroit@hour-20130930134601 pool/detroit@hour-20130930144601\r\n\r\nLine marked with ----> is where it went south.  Note the subsequent snapshot and send operations of \"pool/detroit\" and \"arch/archive\" had no errors...\r\n\r\nIf I can provide more information, please let me know.  I'm an engineer / IT guy not a developer, but I can read and follow instructions well.  =)  I have purposefully left the system as-is.  It is non-primary storage, so it can be rebooted / recompiled / whatever as needed.  Looking forward to hear from you!\r\n\r\nRegards,\r\nOwen L. Wieck\r\nRicardo, Inc."}}