{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4023","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4023/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4023/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4023/events","html_url":"https://github.com/zfsonlinux/zfs/issues/4023","id":117697330,"number":4023,"title":"partial overwrite of metadata holes results in loss of hole birth info","user":{"login":"bprotopopov","id":2488505,"avatar_url":"https://avatars.githubusercontent.com/u/2488505?v=3","gravatar_id":"","url":"https://api.github.com/users/bprotopopov","html_url":"https://github.com/bprotopopov","followers_url":"https://api.github.com/users/bprotopopov/followers","following_url":"https://api.github.com/users/bprotopopov/following{/other_user}","gists_url":"https://api.github.com/users/bprotopopov/gists{/gist_id}","starred_url":"https://api.github.com/users/bprotopopov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprotopopov/subscriptions","organizations_url":"https://api.github.com/users/bprotopopov/orgs","repos_url":"https://api.github.com/users/bprotopopov/repos","events_url":"https://api.github.com/users/bprotopopov/events{/privacy}","received_events_url":"https://api.github.com/users/bprotopopov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-11-18T22:31:31Z","updated_at":"2015-11-19T02:52:58Z","closed_at":null,"body":"When an Ln (n>0) hole is partially overwritten, the remaining (Ln-1) holes do not inherit the Ln's hole's birth epoch. Other properties, such as dmu type, level, and logical size, are not set properly either. In fact, the block pointers for the Ln-1 holes are completely zeroed out. \r\n\r\nAs a result, when the hole_birth feature is active, an incremental zfs send stream will not consider those Ln-1 holes \"new\", and therefore will not include the corresponding FREE records in the stream. This, in turn, results in data loss at the site of the stream receipt. \r\n\r\nLet's consider the following use case. Snapshot s1 has a non-zero L1 block range (say block 128-256). After a number of I/O operations, this range is zeroed out, and it is now represented by a single L1 hole. Then this L1 hole is partly overwritten, resulting in a collection of \"old\" (birth epoch zero) holes in the range under discussion. Then snapshot s2 is created, and zfs send -i s1 s2 is generated. \r\n\r\nIf the hole_birth feature is active, the resulting stream will not contain the FREE records for the holes in the block range in question, and therefore, when the stream is received, the corresponding data blocks will not be zeroed out (data loss).\r\n\r\nHere is an example of this case illustrated with zdb dumps.\r\n\r\nBefore the overwrite, snapshot s1 has an L1 hole:\r\n\r\n          7c0000   L0 1:2584800:10000 10000L/10000P F=1 B=346/346\r\n          7d0000   L0 1:2594800:10000 10000L/10000P F=1 B=346/346\r\n          7e0000   L0 1:25a4800:10000 10000L/10000P F=1 B=346/346\r\n          7f0000   L0 1:25b4800:10000 10000L/10000P F=1 B=346/346\r\n          800000  L1  4000L B=548\r\n         1000000  L1  0:7c23e00:400 1:7c2d200:400 4000L/400P F=128 B=1268/1268\r\n         1000000   L0 0:7808c00:10000 10000L/10000P F=1 B=1268/1268\r\n         1010000   L0 0:7818c00:10000 10000L/10000P F=1 B=1268/1268\r\n         1020000   L0 0:7828c00:10000 10000L/10000P F=1 B=1268/1268\r\n\r\nThe L1 hole is at offset 800000. After a partial overwrite (10 blocks written at the beginning of the L1 range):\r\n\r\n          7d0000   L0 1:2594800:10000 10000L/10000P F=1 B=346/346\r\n          7e0000   L0 1:25a4800:10000 10000L/10000P F=1 B=346/346\r\n          7f0000   L0 1:25b4800:10000 10000L/10000P F=1 B=346/346\r\n          800000  L1  0:ea36000:600 1:f2d6000:600 4000L/600P F=10 B=1749/1749\r\n          800000   L0 1:f246000:10000 10000L/10000P F=1 B=1749/1749\r\n          810000   L0 1:f256000:10000 10000L/10000P F=1 B=1749/1749\r\n          820000   L0 1:f266000:10000 10000L/10000P F=1 B=1749/1749\r\n          830000   L0 1:f276000:10000 10000L/10000P F=1 B=1749/1749\r\n          840000   L0 1:f286000:10000 10000L/10000P F=1 B=1749/1749\r\n          850000   L0 1:f296000:10000 10000L/10000P F=1 B=1749/1749\r\n          860000   L0 1:f2a6000:10000 10000L/10000P F=1 B=1749/1749\r\n          870000   L0 1:f2c6000:10000 10000L/10000P F=1 B=1749/1749\r\n          880000   L0 1:f2b6000:10000 10000L/10000P F=1 B=1749/1749\r\n          890000   L0 0:ea26000:10000 10000L/10000P F=1 B=1749/1749\r\n         1000000  L1  0:7c23e00:400 1:7c2d200:400 4000L/400P F=128 B=1268/1268\r\n         1000000   L0 0:7808c00:10000 10000L/10000P F=1 B=1268/1268\r\n\r\nDump of the new L1 block's contents:\r\nzdb tpool -R 0:ea36000:600:di\r\n```\r\nFound vdev: /dev/sdk1\r\nDVA[0]=<1:f246000:10000> [L0 zvol object] fletcher4 uncompressed LE contiguous unique single size=10000L/10000P birth=1749L/1749P fill=1 cksum=2041f382f58d:408f994ef048de7:daf0e1cadf74f53:47c3fdb952a1e13f\r\nDVA[0]=<1:f256000:10000> [L0 zvol object] fletcher4 uncompressed LE contiguous unique single size=10000L/10000P birth=1749L/1749P fill=1 cksum=20306f1938d5:403381ccd468d8a:713a193137858160:d91b1c5cecb306af\r\nDVA[0]=<1:f266000:10000> [L0 zvol object] fletcher4 uncompressed LE contiguous unique single size=10000L/10000P birth=1749L/1749P fill=1 cksum=1fe160444ab9:3fcbaeb1f31c86e:11198655b490d3b5:76cd3d278385af3e\r\nDVA[0]=<1:f276000:10000> [L0 zvol object] fletcher4 uncompressed LE contiguous unique single size=10000L/10000P birth=1749L/1749P fill=1 cksum=201cb0f386db:4035cd7deb41749:6b4d734a11ce04b5:1fbc2dc2f169dcae\r\nDVA[0]=<1:f286000:10000> [L0 zvol object] fletcher4 uncompressed LE contiguous unique single size=10000L/10000P birth=1749L/1749P fill=1 cksum=202d87c1a695:403d7ff6a1a6f53:66b049fa47216fb4:848b133855fab5b\r\nDVA[0]=<1:f296000:10000> [L0 zvol object] fletcher4 uncompressed LE contiguous unique single size=10000L/10000P birth=1749L/1749P fill=1 cksum=200db48ae914:40163392a6e1f2a:62ad5c6b01c39d36:b1fa1b14d986fa82\r\nDVA[0]=<1:f2a6000:10000> [L0 zvol object] fletcher4 uncompressed LE contiguous unique single size=10000L/10000P birth=1749L/1749P fill=1 cksum=1feb72709d3a:3f8dbd7a3f1e98f:ab207f926cc8b2fc:c9a1145f06e1f9ab\r\nDVA[0]=<1:f2c6000:10000> [L0 zvol object] fletcher4 uncompressed LE contiguous unique single size=10000L/10000P birth=1749L/1749P fill=1 cksum=1fd90de96ff0:3f98dc852f15900:a3cd2aed016bc0a9:eb9f507ffe495f15\r\nDVA[0]=<1:f2b6000:10000> [L0 zvol object] fletcher4 uncompressed LE contiguous unique single size=10000L/10000P birth=1749L/1749P fill=1 cksum=20361039de8d:40b6d13e4438295:75686dbb7da50937:3217ceae84d5b538\r\nDVA[0]=<0:ea26000:10000> [L0 zvol object] fletcher4 uncompressed LE contiguous unique single size=10000L/10000P birth=1749L/1749P fill=1 cksum=202a397fecd2:405130f54a9a83d:cda024b471659627:edf740c0ca1563eb\r\nHOLE [L0 unallocated] size=200L birth=0L\r\nHOLE [L0 unallocated] size=200L birth=0L\r\nHOLE [L0 unallocated] size=200L birth=0L\r\n...\r\nHOLE [L0 unallocated] size=200L birth=0L\r\nHOLE [L0 unallocated] size=200L birth=0L\r\nHOLE [L0 unallocated] size=200L birth=0L\r\n```\r\nThe uncompressed data is likely due to the /dev/urandom source. The volume does have lz4 compression set (and had before the overwrite - inherited from the pool).\r\n\r\nThe new L0 holes have birth epoch 0, instead of 548, as in the original L1 hole."},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/157933100","html_url":"https://github.com/zfsonlinux/zfs/issues/4023#issuecomment-157933100","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/4023","id":157933100,"user":{"login":"bprotopopov","id":2488505,"avatar_url":"https://avatars.githubusercontent.com/u/2488505?v=3","gravatar_id":"","url":"https://api.github.com/users/bprotopopov","html_url":"https://github.com/bprotopopov","followers_url":"https://api.github.com/users/bprotopopov/followers","following_url":"https://api.github.com/users/bprotopopov/following{/other_user}","gists_url":"https://api.github.com/users/bprotopopov/gists{/gist_id}","starred_url":"https://api.github.com/users/bprotopopov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprotopopov/subscriptions","organizations_url":"https://api.github.com/users/bprotopopov/orgs","repos_url":"https://api.github.com/users/bprotopopov/repos","events_url":"https://api.github.com/users/bprotopopov/events{/privacy}","received_events_url":"https://api.github.com/users/bprotopopov/received_events","type":"User","site_admin":false},"created_at":"2015-11-19T02:52:58Z","updated_at":"2015-11-19T02:52:58Z","body":"This issue is similar to \r\nhttps://www.illumos.org/issues/5695 dmu_sync'ed holes do not retain birth time"}}