{"action":"created","issue":{"url":"https://api.github.com/repos/openzfsonosx/zfs/issues/292","labels_url":"https://api.github.com/repos/openzfsonosx/zfs/issues/292/labels{/name}","comments_url":"https://api.github.com/repos/openzfsonosx/zfs/issues/292/comments","events_url":"https://api.github.com/repos/openzfsonosx/zfs/issues/292/events","html_url":"https://github.com/openzfsonosx/zfs/issues/292","id":65684864,"number":292,"title":"ARC Duelling with OS X File Cache","user":{"login":"jawbroken","id":2789,"avatar_url":"https://avatars.githubusercontent.com/u/2789?v=3","gravatar_id":"","url":"https://api.github.com/users/jawbroken","html_url":"https://github.com/jawbroken","followers_url":"https://api.github.com/users/jawbroken/followers","following_url":"https://api.github.com/users/jawbroken/following{/other_user}","gists_url":"https://api.github.com/users/jawbroken/gists{/gist_id}","starred_url":"https://api.github.com/users/jawbroken/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jawbroken/subscriptions","organizations_url":"https://api.github.com/users/jawbroken/orgs","repos_url":"https://api.github.com/users/jawbroken/repos","events_url":"https://api.github.com/users/jawbroken/events{/privacy}","received_events_url":"https://api.github.com/users/jawbroken/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2015-04-01T13:31:29Z","updated_at":"2015-04-10T08:39:28Z","closed_at":"2015-04-01T13:39:03Z","body":"The following report is based on the 1.3.1-RC5 code, but I don't see anything in the final 1.3.1 release that would obviously lead to different behaviour.\r\n\r\nI believe I'm seeing degenerate ARC behaviour as a result of the OS X file cache (as seen in the Memory tab of Activity Monitor) duelling with ARC. As I understand it, both caches are supposed to opportunistically use free memory to cache disk reads, but as neither is aware of the other this results in unstable behaviour. One cache sees there is memory pressure and frees some space, and the other cache sees that it can expand into this new space, etc, leading to them both seesawing randomly back and forth. In my experience, eventually ARC loses and gets relegated to a few hundred megabytes, which makes the pool uselessly slow (slower, in fact, than even a cold boot with an empty ARC).\r\n\r\nThe only way I've discovered to keep this under control is to periodically clear the OS X file cache (with the purge command line tool). I suspect that a real solution would need make ARC aware of the file cache so it could distinguish between real memory pressure and an overeager file cache.\r\n\r\nI attached a chart of the size and target size from arcstat.pl over a period of about 2 days showing the behaviour I'm seeing. The memory in use by applications was essentially constant throughout this time, and the majority of the 16GB of memory was in use by either ARC or the file cache. I haven't included the file cache size on here, as I'm not aware of a command line tool to display it, but monitoring Activity Monitor showed that it will basically be a flipped version of the ARC size. You can see the section at the end where the ARC size became so small that I had to reboot in order to make the pool usable again.\r\n\r\n![screen shot 2015-04-02 at 24 25 53](https://cloud.githubusercontent.com/assets/2789/6941854/ed4a37bc-d8ce-11e4-95fb-72dcec1c2532.png)\r\n"},"comment":{"url":"https://api.github.com/repos/openzfsonosx/zfs/issues/comments/91479040","html_url":"https://github.com/openzfsonosx/zfs/issues/292#issuecomment-91479040","issue_url":"https://api.github.com/repos/openzfsonosx/zfs/issues/292","id":91479040,"user":{"login":"jawbroken","id":2789,"avatar_url":"https://avatars.githubusercontent.com/u/2789?v=3","gravatar_id":"","url":"https://api.github.com/users/jawbroken","html_url":"https://github.com/jawbroken","followers_url":"https://api.github.com/users/jawbroken/followers","following_url":"https://api.github.com/users/jawbroken/following{/other_user}","gists_url":"https://api.github.com/users/jawbroken/gists{/gist_id}","starred_url":"https://api.github.com/users/jawbroken/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jawbroken/subscriptions","organizations_url":"https://api.github.com/users/jawbroken/orgs","repos_url":"https://api.github.com/users/jawbroken/repos","events_url":"https://api.github.com/users/jawbroken/events{/privacy}","received_events_url":"https://api.github.com/users/jawbroken/received_events","type":"User","site_admin":false},"created_at":"2015-04-10T08:39:28Z","updated_at":"2015-04-10T08:39:28Z","body":"And here's what happened when I cleared the OS X file cache manually (using purge).\r\n![screen shot 2015-04-10 at 18 37 05](https://cloud.githubusercontent.com/assets/2789/7084447/da6bf4e4-dfb0-11e4-8d35-c1cb53e3e240.png)\r\n"}}