{"action":"created","issue":{"url":"https://api.github.com/repos/markfasheh/duperemove/issues/51","labels_url":"https://api.github.com/repos/markfasheh/duperemove/issues/51/labels{/name}","comments_url":"https://api.github.com/repos/markfasheh/duperemove/issues/51/comments","events_url":"https://api.github.com/repos/markfasheh/duperemove/issues/51/events","html_url":"https://github.com/markfasheh/duperemove/issues/51","id":65419782,"number":51,"title":"duperemove submissions ignored by kernel except changes modify and change times on files submitted for dedup","user":{"login":"dominicraf","id":177930,"avatar_url":"https://avatars.githubusercontent.com/u/177930?v=3","gravatar_id":"","url":"https://api.github.com/users/dominicraf","html_url":"https://github.com/dominicraf","followers_url":"https://api.github.com/users/dominicraf/followers","following_url":"https://api.github.com/users/dominicraf/following{/other_user}","gists_url":"https://api.github.com/users/dominicraf/gists{/gist_id}","starred_url":"https://api.github.com/users/dominicraf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dominicraf/subscriptions","organizations_url":"https://api.github.com/users/dominicraf/orgs","repos_url":"https://api.github.com/users/dominicraf/repos","events_url":"https://api.github.com/users/dominicraf/events{/privacy}","received_events_url":"https://api.github.com/users/dominicraf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":20,"created_at":"2015-03-31T10:30:45Z","updated_at":"2015-07-04T17:47:58Z","closed_at":null,"body":"using duperemove v0.09.1 with kernel 3.19.3 and options -rdhv I find that after a very long run with many many matched extents found, almost no space is released, presumably because the kernel refuses to honour the requests from duperemove - why? Volume is mounted with options `defaults,compress=lzo,autodefrag` - so compression is used but I believe many of the files refused for deduplication were not in fact compressed.\r\n\r\nEven more seriously, all files identified by `duperemove` as having matched extents (except for one file in each case - the first listed by `duperemove -v` I think) have their modify and change times updated. Is this by design? AFAIK the files have not changed in any way, and they have not even been deduped, but the whole system is now wrecked for rsync'ing to non-btrfs fs.\r\n\r\nExample extract output from duperemove:\r\n\r\n`/home/root-samba/archives/Litos_film.mov       start block: 0 (0)`\r\n`/home/root-samba/archives/rdiff-backup-data/increments/Litos_film.mov.2009-06-03T22:15:03+01:00.snapshot  start block: 0 (0)`\r\n`Dedupe 1 extents with target: (0.0, 225.4M), \"/home/root-samba/archives/Litos_film.mov\"`\r\n`[...]`\r\n`Kernel processed data (excludes target files): 43.0G`\r\n`Comparison of extent info shows a net change in shared extents of: 128.0K`\r\n\r\nChecking afterwards:\r\n\r\n`$ sudo stat \"/home/root-samba/archives/Litos_film.mov\" \"/home/root-samba/archives/rdiff-backup-data/increments/Litos_film.mov.2009-06-03T22:15:03+01:00.snapshot\"`\r\n`  File: ‘/home/root-samba/archives/Litos_film.mov’`\r\n`  Size: 236343878       Blocks: 461616     IO Block: 4096   regular file`\r\n`Device: 1ah/26d Inode: 14033281    Links: 1`\r\n`Access: (0770/-rwxrwx---)  Uid: ( 1003/dominic-pcchips2)   Gid: (  100/   users)`\r\n`Access: 2015-03-29 14:42:28.025850060 +0100`\r\n`Modify: 2006-12-05 12:31:00.000000000 +0000`\r\n`Change: 2015-02-26 18:27:13.732004959 +0000`\r\n` Birth: -`\r\n`  File: ‘/home/root-samba/archives/rdiff-backup-data/increments/Litos_film.mov.2009-06-03T22:15:03+01:00.snapshot’`\r\n`  Size: 236343878       Blocks: 461616     IO Block: 4096   regular file`\r\n`Device: 1ah/26d Inode: 15861510    Links: 1`\r\n`Access: (0770/-rwxrwx---)  Uid: ( 1008/root-samba)   Gid: ( 1008/root-samba)`\r\n`Access: 2015-03-30 06:49:23.265361050 +0100`\r\n`Modify: 2015-03-30 12:57:55.745174664 +0100`\r\n`Change: 2015-03-30 12:57:55.745174664 +0100`\r\n` Birth: -`\r\nThe 2nd file now has modify/change date of 2015-03-30 but in fact has not been changed - except by duperemove - for more than five years."},"comment":{"url":"https://api.github.com/repos/markfasheh/duperemove/issues/comments/118537125","html_url":"https://github.com/markfasheh/duperemove/issues/51#issuecomment-118537125","issue_url":"https://api.github.com/repos/markfasheh/duperemove/issues/51","id":118537125,"user":{"login":"markfasheh","id":2341925,"avatar_url":"https://avatars.githubusercontent.com/u/2341925?v=3","gravatar_id":"","url":"https://api.github.com/users/markfasheh","html_url":"https://github.com/markfasheh","followers_url":"https://api.github.com/users/markfasheh/followers","following_url":"https://api.github.com/users/markfasheh/following{/other_user}","gists_url":"https://api.github.com/users/markfasheh/gists{/gist_id}","starred_url":"https://api.github.com/users/markfasheh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markfasheh/subscriptions","organizations_url":"https://api.github.com/users/markfasheh/orgs","repos_url":"https://api.github.com/users/markfasheh/repos","events_url":"https://api.github.com/users/markfasheh/events{/privacy}","received_events_url":"https://api.github.com/users/markfasheh/received_events","type":"User","site_admin":false},"created_at":"2015-07-04T17:47:58Z","updated_at":"2015-07-04T17:47:58Z","body":"Bedup is using the clone ioctl directly. Duperemove uses the dedupe ioctl which uses the clone code as well. The difference is that dedupe was rejecting some unaligned extent lengths whereas clone would be fine with those. So that explains why bedup worked for you. Basically dedupe was too strict with what it would accept, whereas clone is pretty liberal.\r\n\r\nThe downside is that clone is destructive in that it won't check the targets data. So if you race writing some data to a file that you're cloning to, you could lose that data. The dedupe ioctl checks file data before allowing the extent to be overwritten, therefore you do not have that possibility of losing data.\r\n\r\nI don't want to emulate bedups behavior in duperemove because one promise we make is that dedupe won't (to our knowledge) corrupt user data and the dedupe ioctl is the only way to be sure of that.\r\n\r\nAs far as when you'll get the patches (which I think is the real question here), once they're all in Linus tree (or I'm convinced they'll wind up there) I'd like to make them available, with description in the duperemove tree so it's easy for people who compile their own kernel to get them. I'll also try to get the most important ones backported to the older stable kernel releases.\r\n\r\nLastly if you're running SUSE or openSUSE I can have a more direct role in getting the patches into those kernels so you can expect them their sooner rather than later. That isn't an advertisment or anything btw, I just happen to work there so I can push the patches myself."}}