{"action":"created","issue":{"url":"https://api.github.com/repos/jmacd/xdelta/issues/196","labels_url":"https://api.github.com/repos/jmacd/xdelta/issues/196/labels{/name}","comments_url":"https://api.github.com/repos/jmacd/xdelta/issues/196/comments","events_url":"https://api.github.com/repos/jmacd/xdelta/issues/196/events","html_url":"https://github.com/jmacd/xdelta/issues/196","id":85721492,"number":196,"title":"Source buffer size","user":{"login":"hurda","id":3539609,"avatar_url":"https://avatars.githubusercontent.com/u/3539609?v=3","gravatar_id":"","url":"https://api.github.com/users/hurda","html_url":"https://github.com/hurda","followers_url":"https://api.github.com/users/hurda/followers","following_url":"https://api.github.com/users/hurda/following{/other_user}","gists_url":"https://api.github.com/users/hurda/gists{/gist_id}","starred_url":"https://api.github.com/users/hurda/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hurda/subscriptions","organizations_url":"https://api.github.com/users/hurda/orgs","repos_url":"https://api.github.com/users/hurda/repos","events_url":"https://api.github.com/users/hurda/events{/privacy}","received_events_url":"https://api.github.com/users/hurda/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-06-06T08:10:00Z","updated_at":"2015-06-09T01:08:16Z","closed_at":null,"body":"Using a commandline\r\n`xdelta3-3.0.8.x86-64.exe -e -I 0 -B 268435456 -P 16777216 -W 16777216`\r\nfor comparing two 2GB files, xdelta is allocating memory the size of 2*(source buffer size) (-B), but only finding differences when they're less or equal than (source buffer size)/2 apart.\r\nFor my commandline this means it's allocating a total of 512MB of memory, and finds differences (and similarities) in a 128MB-range.\r\n\r\n\r\n>To ensure the source input is read sequentially, with no backward seeks, the encoder \r\n>maintains the source horizon at half the source buffer size ahead of the input position. \r\n>A source copy will not be found if it lies more than half the source buffer size away \r\n>from its absolute position in the input stream. \r\n\r\nhttp://web.archive.org/web/20140709015104/http://code.google.com/p/xdelta/wiki/TuningMemoryBudget\r\n\r\nSo my question is: What's happening with the other half? What is it used for?"},"comment":{"url":"https://api.github.com/repos/jmacd/xdelta/issues/comments/110187521","html_url":"https://github.com/jmacd/xdelta/issues/196#issuecomment-110187521","issue_url":"https://api.github.com/repos/jmacd/xdelta/issues/196","id":110187521,"user":{"login":"jmacd","id":3629705,"avatar_url":"https://avatars.githubusercontent.com/u/3629705?v=3","gravatar_id":"","url":"https://api.github.com/users/jmacd","html_url":"https://github.com/jmacd","followers_url":"https://api.github.com/users/jmacd/followers","following_url":"https://api.github.com/users/jmacd/following{/other_user}","gists_url":"https://api.github.com/users/jmacd/gists{/gist_id}","starred_url":"https://api.github.com/users/jmacd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmacd/subscriptions","organizations_url":"https://api.github.com/users/jmacd/orgs","repos_url":"https://api.github.com/users/jmacd/repos","events_url":"https://api.github.com/users/jmacd/events{/privacy}","received_events_url":"https://api.github.com/users/jmacd/received_events","type":"User","site_admin":false},"created_at":"2015-06-09T01:08:16Z","updated_at":"2015-06-09T01:08:16Z","body":"For your first question, the other memory goes to a very large hash table\ndata structure.\nYour second question -- sounds unusual but you are still passing the -B\nflag with 256MB right?\n\nThere are missed opportunities to rduve memory usage in case a like this,\nbut they complicate the code quite a bit. The code is organized to work\nwith streams, including compressed ones, which makes these memory\noptimizations really tricky since you don't know the size of a stream up\nfront. I removed a few memory-reducing special cases a few releases ago...\nOn Jun 6, 2015 4:10 AM, \"hurda\" <notifications@github.com> wrote:\n\n> Using a commandline\n> xdelta3-3.0.8.x86-64.exe -e -I 0 -B 268435456 -P 16777216 -W 16777216\n> for comparing two 2GB files, xdelta is allocating memory the size of\n> 2*(source buffer size) (-B), but only finding differences when they're less\n> or equal than (source buffer size)/2 apart.\n> For my commandline this means it's allocating a total of 512MB of memory,\n> and finds differences (and similarities) in a 128MB-range.\n>\n> To ensure the source input is read sequentially, with no backward seeks,\n> the encoder\n> maintains the source horizon at half the source buffer size ahead of the\n> input position.\n> A source copy will not be found if it lies more than half the source\n> buffer size away\n> from its absolute position in the input stream.\n>\n>\n> http://web.archive.org/web/20140709015104/http://code.google.com/p/xdelta/wiki/TuningMemoryBudget\n>\n> So my question is: What's happening with the other half? What is it used\n> for?\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/jmacd/xdelta/issues/196>.\n>\n"}}