{"action":"opened","issue":{"url":"https://api.github.com/repos/BOINC/boinc-ng/issues/864","labels_url":"https://api.github.com/repos/BOINC/boinc-ng/issues/864/labels{/name}","comments_url":"https://api.github.com/repos/BOINC/boinc-ng/issues/864/comments","events_url":"https://api.github.com/repos/BOINC/boinc-ng/issues/864/events","html_url":"https://github.com/BOINC/boinc-ng/issues/864","id":56473365,"number":864,"title":"Serious sample_bitwise_validator issues with binary files","user":{"login":"romw","id":985772,"avatar_url":"https://avatars.githubusercontent.com/u/985772?v=3","gravatar_id":"","url":"https://api.github.com/users/romw","html_url":"https://github.com/romw","followers_url":"https://api.github.com/users/romw/followers","following_url":"https://api.github.com/users/romw/following{/other_user}","gists_url":"https://api.github.com/users/romw/gists{/gist_id}","starred_url":"https://api.github.com/users/romw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/romw/subscriptions","organizations_url":"https://api.github.com/users/romw/orgs","repos_url":"https://api.github.com/users/romw/repos","events_url":"https://api.github.com/users/romw/events{/privacy}","received_events_url":"https://api.github.com/users/romw/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/BOINC/boinc-ng/labels/C:%20Server%20-%20Validator","name":"C: Server - Validator","color":"0000aa"},{"url":"https://api.github.com/repos/BOINC/boinc-ng/labels/P:%20Critical","name":"P: Critical","color":"ffffb8"},{"url":"https://api.github.com/repos/BOINC/boinc-ng/labels/R:%20fixed","name":"R: fixed","color":"55ff55"},{"url":"https://api.github.com/repos/BOINC/boinc-ng/labels/T:%20Defect","name":"T: Defect","color":"cccccc"}],"state":"open","locked":false,"assignee":{"login":"davidpanderson","id":985729,"avatar_url":"https://avatars.githubusercontent.com/u/985729?v=3","gravatar_id":"","url":"https://api.github.com/users/davidpanderson","html_url":"https://github.com/davidpanderson","followers_url":"https://api.github.com/users/davidpanderson/followers","following_url":"https://api.github.com/users/davidpanderson/following{/other_user}","gists_url":"https://api.github.com/users/davidpanderson/gists{/gist_id}","starred_url":"https://api.github.com/users/davidpanderson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidpanderson/subscriptions","organizations_url":"https://api.github.com/users/davidpanderson/orgs","repos_url":"https://api.github.com/users/davidpanderson/repos","events_url":"https://api.github.com/users/davidpanderson/events{/privacy}","received_events_url":"https://api.github.com/users/davidpanderson/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/BOINC/boinc-ng/milestones/3","labels_url":"https://api.github.com/repos/BOINC/boinc-ng/milestones/3/labels","id":964209,"number":3,"title":"Undetermined","description":"Features for this release:\r\n\r\n * WIN: Streamline installer for mass deployments using either Installshield or WiX\r\n","creator":{"login":"romw","id":985772,"avatar_url":"https://avatars.githubusercontent.com/u/985772?v=3","gravatar_id":"","url":"https://api.github.com/users/romw","html_url":"https://github.com/romw","followers_url":"https://api.github.com/users/romw/followers","following_url":"https://api.github.com/users/romw/following{/other_user}","gists_url":"https://api.github.com/users/romw/gists{/gist_id}","starred_url":"https://api.github.com/users/romw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/romw/subscriptions","organizations_url":"https://api.github.com/users/romw/orgs","repos_url":"https://api.github.com/users/romw/repos","events_url":"https://api.github.com/users/romw/events{/privacy}","received_events_url":"https://api.github.com/users/romw/received_events","type":"User","site_admin":false},"open_issues":138,"closed_issues":537,"state":"open","created_at":"2015-02-03T19:47:35Z","updated_at":"2015-02-04T02:58:30Z","due_on":"1970-01-01T00:00:00Z","closed_at":null},"comments":0,"created_at":"2015-02-04T02:58:30Z","updated_at":"2015-02-04T02:58:30Z","closed_at":null,"body":"**Reported by Nicolas on 19 Dec 39331775 02:13 UTC**\n```sample_bitwise_validator``` works by calculating MD5 hashes of the files to be compared, then comparing the hashes, not the file data itself.\r\n\r\nTo calculate the hash, it loads the whole file in memory, calls ```md5_string``` on the data, then deletes the data.\r\n\r\nThe ```read_file_string``` function it uses to read the file needs file_size*2 RAM and '''truncates the data at the first null byte'''. See #886 for that problem.\r\n\r\nThe result is that if two files are overall different, but identical in the data before first null byte, the validator will say they match. I tested, for example, that it considers all object files (```.o```) and all linked binaries of the server code as identical. I tested it on six zip archives with completely different contents, and all except one were marked as valid to each other, which is perhaps more worrying (because projects actually use those).\r\n\r\nWhile fixing #886 will also fix the binary file issue here, it will still load the whole file in memory. I attach a patch that uses ```md5_file``` instead, making it have O(1) memory usage, and fixing this problem independently of #886.\r\n\n\nMigrated-From: http://boinc.berkeley.edu/trac/ticket/887"}}