{"action":"created","issue":{"url":"https://api.github.com/repos/quixdb/squash/issues/120","labels_url":"https://api.github.com/repos/quixdb/squash/issues/120/labels{/name}","comments_url":"https://api.github.com/repos/quixdb/squash/issues/120/comments","events_url":"https://api.github.com/repos/quixdb/squash/issues/120/events","html_url":"https://github.com/quixdb/squash/issues/120","id":106826672,"number":120,"title":"map files when possible for I/O funcs","user":{"login":"nemequ","id":1151,"avatar_url":"https://avatars.githubusercontent.com/u/1151?v=3","gravatar_id":"","url":"https://api.github.com/users/nemequ","html_url":"https://github.com/nemequ","followers_url":"https://api.github.com/users/nemequ/followers","following_url":"https://api.github.com/users/nemequ/following{/other_user}","gists_url":"https://api.github.com/users/nemequ/gists{/gist_id}","starred_url":"https://api.github.com/users/nemequ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nemequ/subscriptions","organizations_url":"https://api.github.com/users/nemequ/orgs","repos_url":"https://api.github.com/users/nemequ/repos","events_url":"https://api.github.com/users/nemequ/events{/privacy}","received_events_url":"https://api.github.com/users/nemequ/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/quixdb/squash/labels/core","name":"core","color":"f7c6c7"},{"url":"https://api.github.com/repos/quixdb/squash/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/quixdb/squash/milestones/5","html_url":"https://github.com/quixdb/squash/milestones/0.8","labels_url":"https://api.github.com/repos/quixdb/squash/milestones/5/labels","id":1303338,"number":5,"title":"0.8","description":"","creator":{"login":"nemequ","id":1151,"avatar_url":"https://avatars.githubusercontent.com/u/1151?v=3","gravatar_id":"","url":"https://api.github.com/users/nemequ","html_url":"https://github.com/nemequ","followers_url":"https://api.github.com/users/nemequ/followers","following_url":"https://api.github.com/users/nemequ/following{/other_user}","gists_url":"https://api.github.com/users/nemequ/gists{/gist_id}","starred_url":"https://api.github.com/users/nemequ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nemequ/subscriptions","organizations_url":"https://api.github.com/users/nemequ/orgs","repos_url":"https://api.github.com/users/nemequ/repos","events_url":"https://api.github.com/users/nemequ/events{/privacy}","received_events_url":"https://api.github.com/users/nemequ/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":0,"state":"open","created_at":"2015-09-13T22:26:38Z","updated_at":"2015-09-16T17:58:00Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2015-09-16T17:58:00Z","updated_at":"2015-09-19T07:28:32Z","closed_at":null,"body":"Instead of allocating a buffer to compress/decompress to then writing/reading that to the file with fwrite/fread, we should try to use mmap.  Should provide a performance improvement."},"comment":{"url":"https://api.github.com/repos/quixdb/squash/issues/comments/141632969","html_url":"https://github.com/quixdb/squash/issues/120#issuecomment-141632969","issue_url":"https://api.github.com/repos/quixdb/squash/issues/120","id":141632969,"user":{"login":"nemequ","id":1151,"avatar_url":"https://avatars.githubusercontent.com/u/1151?v=3","gravatar_id":"","url":"https://api.github.com/users/nemequ","html_url":"https://github.com/nemequ","followers_url":"https://api.github.com/users/nemequ/followers","following_url":"https://api.github.com/users/nemequ/following{/other_user}","gists_url":"https://api.github.com/users/nemequ/gists{/gist_id}","starred_url":"https://api.github.com/users/nemequ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nemequ/subscriptions","organizations_url":"https://api.github.com/users/nemequ/orgs","repos_url":"https://api.github.com/users/nemequ/repos","events_url":"https://api.github.com/users/nemequ/events{/privacy}","received_events_url":"https://api.github.com/users/nemequ/received_events","type":"User","site_admin":false},"created_at":"2015-09-19T07:28:32Z","updated_at":"2015-09-19T07:28:32Z","body":"> Mmap often makes IO faster, but not always. In particular it depends a lot of the underlying OS implementation of read-ahead. With read() and write() you can explicitly read large chunks, but to get the same effect with mmap the OS has to be smart about it (or you have to use madavise or similar). In my experience recent Linux kernels do a good job.\r\n\r\nYep, it needs benchmarking.  I actually wrote most of the code for this a few days ago, but it is [#ifdefed out](https://github.com/quixdb/squash/blob/master/squash/file.c#L41) until I have some time to test performance.\r\n\r\n> Older Windows versions not so much - only as an example.\r\n\r\nIt would always be possible to just `#ifdef` out the mapping code on Windows.\r\n\r\n> It's also worth noting that switching to memory mapping means that it's tougher to separate the IO from the CPU processing, for example for benchmarking purposes. Mmap always inherently interleaves with the normal operations of compression engine. The ideal access pattern into the input buffer may change with mmap!\r\n\r\nNot really; well, at least not for us.  It's not like there are perf counters in Squash which are toggled on and off—the benchmark just [lets the OS handle the details](https://github.com/quixdb/squash-benchmark/blob/master/timer.c).  I/O time doesn't count against CPU time regardless of whether you're using mmap or stdio (except a small amount of time for both when the CPU is actually working, but it's pretty minimal for both).\r\n\r\n> Have you considered a mode where the IO happens in another thread, overlapped with the CPU compression work? Perhaps useful for real world use of the squash CLI.\r\n\r\nNot really.  I'd be willing to consider it, but spinning up a thread is a pretty big performance hit.  You might see good results for larger files, but it would kill performance for small buffers.  I think you would needs some heuristics for deciding whether or not it is worth it to spin up a thread, and it would only work for regular files (you don't know the size of a pipe, for example).  My gut feeling is that it would probably be better to just mmap reasonably large chunks and let the OS handle shuffling data around.  That's basically the whole idea behind [LMDB](http://symas.com/mdb/), and I can personally vouch for the fact that LMDB works pretty well."}}