{"action":"created","issue":{"url":"https://api.github.com/repos/axboe/fio/issues/48","labels_url":"https://api.github.com/repos/axboe/fio/issues/48/labels{/name}","comments_url":"https://api.github.com/repos/axboe/fio/issues/48/comments","events_url":"https://api.github.com/repos/axboe/fio/issues/48/events","html_url":"https://github.com/axboe/fio/issues/48","id":58504014,"number":48,"title":"OSX fcntl(fd, F_NOCACHE, 1) not equivalent to O_DIRECT on Linux","user":{"login":"aktau","id":189413,"avatar_url":"https://avatars.githubusercontent.com/u/189413?v=3","gravatar_id":"","url":"https://api.github.com/users/aktau","html_url":"https://github.com/aktau","followers_url":"https://api.github.com/users/aktau/followers","following_url":"https://api.github.com/users/aktau/following{/other_user}","gists_url":"https://api.github.com/users/aktau/gists{/gist_id}","starred_url":"https://api.github.com/users/aktau/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aktau/subscriptions","organizations_url":"https://api.github.com/users/aktau/orgs","repos_url":"https://api.github.com/users/aktau/repos","events_url":"https://api.github.com/users/aktau/events{/privacy}","received_events_url":"https://api.github.com/users/aktau/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-02-22T14:19:15Z","updated_at":"2015-02-22T16:07:56Z","closed_at":null,"body":"This is probably already well known, but it bit me while planning to do some comparative benchmarks between Linux and OSX. I started with OSX and `direct=1, rw=randread`, but noticed that OSX was reading at 1200MB/s from a 1GB file. This was unexpected as I only have one spinning rust HDD in the MBP. \r\n\r\nSo I looked up the ways to do direct I/O on OSX. On stack overflow and the Apple mailing lists, `fcntl(fd, F_NOCACHE, 1)` looked to be the canonical solution. This was also implemented in fio(1) in 2011 in commit 7e8ad197. It seems that `F_NOCACHE` disables the page cache from that point on, but the file in quesition was _already_ in the page cache, it will not be purged and the pages will be used.\r\n\r\nI also commented on [stack overflow: clear buffer cache on OSX]( http://stackoverflow.com/questions/459537/clear-buffer-cache-on-mac-os-x) with my observations. I'll copy it here as well:\r\n\r\n> It's my impression that even when turning off the cache like this (with F_NOCACHE or F_GLOBAL_NOCACHE), if there are pages of the file already in the page cache, those will still be used. I tried to test this by using fio(1) with direct=1. It seems to confirm my suspicions (I get ~1200MB/s throughput on a random read, on a spindle HDD in my MBP, not an SSD). I've confirmed with dtruss that fio(1) actually calls fcntl correctly.\r\n\r\n> After running \"sudo purge\" and trying the same fio(1) invocation, it's much slower. So yes, it appears that F_NOCACHE is no direct equivalent of O_DIRECT on Linux.\r\n\r\nI'm not advocating running `sudo purge` as part of fio, but perhaps it could be added to the documentation of `direct` that the behaviour is quite different from `O_DIRECT` but they're both fio's way of doing direct IO. Running `sudo purge` is both slow, has an adverse effect on the rest of the system while (and after) it runs for obvious reasons.\r\n\r\nAnother idea I had was to forcibly re-write the file each time for reading, while having the file open with `F_NOCACHE`, which makes the written pages not enter the page cache (UPC). That would also be slow (possibly) but hopefully it wouldn't evict other, unrelated files.\r\n\r\nAn interesting discussion involving an Apple dev on the Apple mailing list: http://lists.apple.com/archives/filesystem-dev/2007/Sep/msg00010.html. Specifically the part [mentioning that files opened with F_NOCACHE will still uses already loaded pages if they're present](http://lists.apple.com/archives/filesystem-dev/2007/Sep/msg00012.html).\r\n\r\n**EDIT**: Some train of thought rambling: but perhaps mmap + MAP_NOCACHE on OSX is a possible avenue (+ telling uses to use that for direct IO testing). The man page doesn't give me a lot of hope though, as it appears that there's not guarantess and the OS might keep things in memory if it feels like it anyway. A small reference mentioning it: http://www.qtcentre.org/threads/24733-High-performance-large-file-reading-on-OSX"},"comment":{"url":"https://api.github.com/repos/axboe/fio/issues/comments/75442374","html_url":"https://github.com/axboe/fio/issues/48#issuecomment-75442374","issue_url":"https://api.github.com/repos/axboe/fio/issues/48","id":75442374,"user":{"login":"aktau","id":189413,"avatar_url":"https://avatars.githubusercontent.com/u/189413?v=3","gravatar_id":"","url":"https://api.github.com/users/aktau","html_url":"https://github.com/aktau","followers_url":"https://api.github.com/users/aktau/followers","following_url":"https://api.github.com/users/aktau/following{/other_user}","gists_url":"https://api.github.com/users/aktau/gists{/gist_id}","starred_url":"https://api.github.com/users/aktau/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aktau/subscriptions","organizations_url":"https://api.github.com/users/aktau/orgs","repos_url":"https://api.github.com/users/aktau/repos","events_url":"https://api.github.com/users/aktau/events{/privacy}","received_events_url":"https://api.github.com/users/aktau/received_events","type":"User","site_admin":false},"created_at":"2015-02-22T16:07:56Z","updated_at":"2015-02-22T16:07:56Z","body":"> Thanks for doing a thorough investigation! \r\n\r\nNo problem at all. I'm glad I learned something and found a workaround in short order. At least the difference was large enough (1200MB/s vs 22MB/s i pretty noticeable).\r\n\r\n> But it really is a horrible interface, seems strange that OSX doesn't give you finer control of what is cached and how.\r\n\r\nI agree. I've looked for a bit more but could find nothing at all. It seems `purge(8)` uses a kernel extension that isn't properly documented. So it doesn't seem productive to find out whether or not it could be used for fio's purposes. A bit of documentation seems like the best option at the moment."}}