{"action":"created","issue":{"url":"https://api.github.com/repos/markfasheh/duperemove/issues/63","labels_url":"https://api.github.com/repos/markfasheh/duperemove/issues/63/labels{/name}","comments_url":"https://api.github.com/repos/markfasheh/duperemove/issues/63/comments","events_url":"https://api.github.com/repos/markfasheh/duperemove/issues/63/events","html_url":"https://github.com/markfasheh/duperemove/issues/63","id":76304722,"number":63,"title":"Output buffer causes empty \"log\" when running on big files in background","user":{"login":"Edoa","id":7404683,"avatar_url":"https://avatars.githubusercontent.com/u/7404683?v=3","gravatar_id":"","url":"https://api.github.com/users/Edoa","html_url":"https://github.com/Edoa","followers_url":"https://api.github.com/users/Edoa/followers","following_url":"https://api.github.com/users/Edoa/following{/other_user}","gists_url":"https://api.github.com/users/Edoa/gists{/gist_id}","starred_url":"https://api.github.com/users/Edoa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Edoa/subscriptions","organizations_url":"https://api.github.com/users/Edoa/orgs","repos_url":"https://api.github.com/users/Edoa/repos","events_url":"https://api.github.com/users/Edoa/events{/privacy}","received_events_url":"https://api.github.com/users/Edoa/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-05-14T09:53:45Z","updated_at":"2015-09-11T14:43:24Z","closed_at":null,"body":"Hi all\r\nI was trying deduperemove on very big files. I use nohup tool to launch long tasks so I can close the terminal without interrupting the task. Then I check nohup.out (where stdout and stderr are redirected by default) from time to time to check the progress.\r\nIn case of duperemove, this file contains just the stderr (e.g. ```Warning: murmur3 support is experimental!```) but not the verbose log that I usually see in my terminal, at least not in a long time.\r\nThe issue doesn't present if I run duperemove in a little folder with little files.\r\n\r\nI found this workaround:\r\n```stdbuf -i0 -o0 -e0 ./duperemove --hashfile=hashes.file -dAv [...] > log.txt```\r\nBut I think we should flush standard output buffer every line or from time to time...\r\n"},"comment":{"url":"https://api.github.com/repos/markfasheh/duperemove/issues/comments/139564246","html_url":"https://github.com/markfasheh/duperemove/issues/63#issuecomment-139564246","issue_url":"https://api.github.com/repos/markfasheh/duperemove/issues/63","id":139564246,"user":{"login":"Edoa","id":7404683,"avatar_url":"https://avatars.githubusercontent.com/u/7404683?v=3","gravatar_id":"","url":"https://api.github.com/users/Edoa","html_url":"https://github.com/Edoa","followers_url":"https://api.github.com/users/Edoa/followers","following_url":"https://api.github.com/users/Edoa/following{/other_user}","gists_url":"https://api.github.com/users/Edoa/gists{/gist_id}","starred_url":"https://api.github.com/users/Edoa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Edoa/subscriptions","organizations_url":"https://api.github.com/users/Edoa/orgs","repos_url":"https://api.github.com/users/Edoa/repos","events_url":"https://api.github.com/users/Edoa/events{/privacy}","received_events_url":"https://api.github.com/users/Edoa/received_events","type":"User","site_admin":false},"created_at":"2015-09-11T14:43:24Z","updated_at":"2015-09-11T14:43:24Z","body":"> I'm not using the nohup progam though, just piping to a file. Maybe it's worse there for some reason?\r\nI don't think so. Maybe it's worse because of the low number of files.\r\n\r\n>I see that you're only doing like 2-4 files when this happens? That's pretty extreme.\r\nThose files are some iSCSI disk images. I've read that duperemove is not optimal on Disk images, but if it worked good, it would be very effective on this kind of data (most parts of the operating system files are the same in different VMs disks)."}}