{"action":"created","issue":{"url":"https://api.github.com/repos/jemalloc/jemalloc/issues/201","labels_url":"https://api.github.com/repos/jemalloc/jemalloc/issues/201/labels{/name}","comments_url":"https://api.github.com/repos/jemalloc/jemalloc/issues/201/comments","events_url":"https://api.github.com/repos/jemalloc/jemalloc/issues/201/events","html_url":"https://github.com/jemalloc/jemalloc/issues/201","id":59985272,"number":201,"title":"Support for user-defined chunk_alloc_mmap/chunk_alloc_munmap","user":{"login":"krzycz","id":8184299,"avatar_url":"https://avatars.githubusercontent.com/u/8184299?v=3","gravatar_id":"","url":"https://api.github.com/users/krzycz","html_url":"https://github.com/krzycz","followers_url":"https://api.github.com/users/krzycz/followers","following_url":"https://api.github.com/users/krzycz/following{/other_user}","gists_url":"https://api.github.com/users/krzycz/gists{/gist_id}","starred_url":"https://api.github.com/users/krzycz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krzycz/subscriptions","organizations_url":"https://api.github.com/users/krzycz/orgs","repos_url":"https://api.github.com/users/krzycz/repos","events_url":"https://api.github.com/users/krzycz/events{/privacy}","received_events_url":"https://api.github.com/users/krzycz/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/jemalloc/jemalloc/labels/feature","name":"feature","color":"207de5"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/jemalloc/jemalloc/milestones/2","html_url":"https://github.com/jemalloc/jemalloc/milestones/4.0.0","labels_url":"https://api.github.com/repos/jemalloc/jemalloc/milestones/2/labels","id":467147,"number":2,"title":"4.0.0","description":"","creator":{"login":"jasone","id":245970,"avatar_url":"https://avatars.githubusercontent.com/u/245970?v=3","gravatar_id":"","url":"https://api.github.com/users/jasone","html_url":"https://github.com/jasone","followers_url":"https://api.github.com/users/jasone/followers","following_url":"https://api.github.com/users/jasone/following{/other_user}","gists_url":"https://api.github.com/users/jasone/gists{/gist_id}","starred_url":"https://api.github.com/users/jasone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jasone/subscriptions","organizations_url":"https://api.github.com/users/jasone/orgs","repos_url":"https://api.github.com/users/jasone/repos","events_url":"https://api.github.com/users/jasone/events{/privacy}","received_events_url":"https://api.github.com/users/jasone/received_events","type":"User","site_admin":false},"open_issues":10,"closed_issues":47,"state":"open","created_at":"2013-10-28T18:59:17Z","updated_at":"2015-03-19T02:00:57Z","due_on":null,"closed_at":null},"comments":6,"created_at":"2015-03-05T17:27:32Z","updated_at":"2015-03-23T23:39:45Z","closed_at":null,"body":"User defined chunk allocators have no access to the extant/free chunk trees, so they can't use the *jemalloc* built-in chunk recycling logic. Moreover, when implementing such functionality in the custom chunk allocator, it's not even possible to re-use some part of *jemalloc* implementation (except by duplicating some *jemalloc* code), as it's all internal.\r\n\r\nFor some applications however, all what the custom allocator is expected to do is to add some additional flags to `mmap()` call (like `MAP_HUGETLB`), set the NUMA memory policy, etc. So, in many cases, instead of replacing the entire `chunk_alloc` path with the custom implementation, it would be enough to substitute only the bottom part of `chunk_alloc` (I mean `chunk_alloc_mmap/chunk_alloc_dss`) with some user-defined function. This would reduce the amount of code required to implement the custom chunk allocator, and it would also allow the custom allocator could still benefit from the chunk recycling mechanism.\r\n\r\nInitially, I was thinking about adding an optional third function to the existing `chunk_alloc_mmap` and `chunk_alloc_dss`, along with its own dedicated set of chunk trees. But after all, I think that it will be easier to allow user to simply override the default implementation of `chunk_alloc_mmap` with its own version. The custom function could then reuse the existing `chunks_szad_mmap/chunks_ad_mmap` trees.\r\nAlso, since the trees are now per arena, then chunks mapped using the custom function would be isolated from the chunks coming from the default *jemalloc* function.\r\n\r\nI believe such functionality won't be in conflict with the existing user-defined `chunk_alloc/chunk_dalloc` feature. It's still up to the user if their prefer to override the entire `chunk_alloc` stuff, or just the bottom part of it.\r\nSeems like it would also address the issue #176.\r\n\r\nThe prototype implementation is available here: https://github.com/krzycz/jemalloc/commit/eb5364b282eb857fa2e6d00df8872356557962bc\r\n\r\nPlease, take a look and let me know what do you think about it.\r\n"},"comment":{"url":"https://api.github.com/repos/jemalloc/jemalloc/issues/comments/85247389","html_url":"https://github.com/jemalloc/jemalloc/issues/201#issuecomment-85247389","issue_url":"https://api.github.com/repos/jemalloc/jemalloc/issues/201","id":85247389,"user":{"login":"cmcantalupo","id":378319,"avatar_url":"https://avatars.githubusercontent.com/u/378319?v=3","gravatar_id":"","url":"https://api.github.com/users/cmcantalupo","html_url":"https://github.com/cmcantalupo","followers_url":"https://api.github.com/users/cmcantalupo/followers","following_url":"https://api.github.com/users/cmcantalupo/following{/other_user}","gists_url":"https://api.github.com/users/cmcantalupo/gists{/gist_id}","starred_url":"https://api.github.com/users/cmcantalupo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cmcantalupo/subscriptions","organizations_url":"https://api.github.com/users/cmcantalupo/orgs","repos_url":"https://api.github.com/users/cmcantalupo/repos","events_url":"https://api.github.com/users/cmcantalupo/events{/privacy}","received_events_url":"https://api.github.com/users/cmcantalupo/received_events","type":"User","site_admin":false},"created_at":"2015-03-23T23:39:45Z","updated_at":"2015-03-23T23:39:45Z","body":"@jasone, Sorry that I may have confused this issue more than I clarified.  I will review the radix tree implementation and get back here about how we may keep virtual address policies separate for huge allocations.  I think it should be straight forward.  \r\n\r\nI have posted an article that we submitted to Runtime and Operating Systems for Supercomputers (ROSS) 2015 here:\r\n\r\nhttp://memkind.github.io/memkind/memkind_arch_20150318.pdf\r\n\r\nThis may serve to motivate the use case for the feature described in this issue, and give you some more background on the approach we are taking.  Thanks for trying to clarify the implementation details.  "}}