{"action":"opened","issue":{"url":"https://api.github.com/repos/rust-lang/rust/issues/30170","labels_url":"https://api.github.com/repos/rust-lang/rust/issues/30170/labels{/name}","comments_url":"https://api.github.com/repos/rust-lang/rust/issues/30170/comments","events_url":"https://api.github.com/repos/rust-lang/rust/issues/30170/events","html_url":"https://github.com/rust-lang/rust/issues/30170","id":120024495,"number":30170,"title":"if huge requested align, alloc_system heap::allocate on OS X returns unaligned values","user":{"login":"pnkfelix","id":173127,"avatar_url":"https://avatars.githubusercontent.com/u/173127?v=3","gravatar_id":"","url":"https://api.github.com/users/pnkfelix","html_url":"https://github.com/pnkfelix","followers_url":"https://api.github.com/users/pnkfelix/followers","following_url":"https://api.github.com/users/pnkfelix/following{/other_user}","gists_url":"https://api.github.com/users/pnkfelix/gists{/gist_id}","starred_url":"https://api.github.com/users/pnkfelix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pnkfelix/subscriptions","organizations_url":"https://api.github.com/users/pnkfelix/orgs","repos_url":"https://api.github.com/users/pnkfelix/repos","events_url":"https://api.github.com/users/pnkfelix/events{/privacy}","received_events_url":"https://api.github.com/users/pnkfelix/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-12-02T20:04:48Z","updated_at":"2015-12-02T20:04:48Z","closed_at":null,"body":"While trying to isolate what kinds of preconditions the `Allocator` trait is going to require and post-conditions it ensures, I made a test program to explore what happens with our current pair of low-level allocators.\r\n\r\nAs far as I can tell so far, jemalloc always ensures that it never gives back an unaligned address.\r\n\r\nbut on OS X, the system allocator (linked via `extern crate alloc_system;`) can produce addresses that do not meet the alignment request, namely if you ask for values aligned to values greater than or equal to `1 << 32`.  (Like i said, we're talking about *huge* alignments here.)\r\n\r\n(on Linux, for both jemalloc and the system allocator, I never observed either returning non-null addresses that did not meet the alignment request.  In other words, they handled absurdly large alignments \"properly\")\r\n\r\n### What to do about this?\r\n\r\nI talk a little about this further in my Digression section below, but in summary, I think we should do *something* about this.\r\n\r\nIt seems like an easy short term solution is this: We should be able to pre-determine which allocator + platform combinations fall victim to this problem, and for those cases, make the `allocate` (and `reallocate`, etc) methods validate their results. If validation fails, then we can just free the original address and return null. \r\n\r\n  * The main problem with this is that it adds overhead to the allocation path.  (Admittedly a divide and branch doesn't seem like much in the context of an allocation, but still...)\r\n\r\nAnyway, if you're interested in alternative ideas, see the Digression/Discussion section below\r\n\r\n### Digression and/or Discussion\r\n\r\nWhen Rust [originally merged][original line] the jemalloc support PR, it stated the following requirement on the `allocate` method:\r\n\r\n```\r\nalignment must be no larger than the largest supported page size on the platform.\r\n```\r\n\r\n[original line]: https://github.com/rust-lang/rust/commit/1b1ca6d5465ef4de12b1adf25cd4598f261c660d#diff-2020572c14fe23a6b11b55b5e113b4aaR36\r\n\r\n  * depending on what the phrase \"the largest supported page size\" is supposed to mean, perhaps even moderately sized values are not legal inputs for the alignment...\r\n  * (was that phrase supposed to be something like \"allocated block size\" rather than \"page size\" ? The notion of more than one memory page size on a given platform is not something I'm familiar with...)\r\n\r\n... so arguably there has always been a precondition to not feed in ridiculously large values for alignment.   \r\n\r\nHowever, even if that is the case, here are some things to consider\r\n\r\n#### Requirements should be checkable\r\n\r\nif we want to continue to impose this precondition, then we *must* provide the programmer with a way to query what the value of \"the largest supported page size on the platform\" actually *is*.\r\n\r\n(I didn't see a way to do this via a couple of greps of the code base, but it doesn't help that I don't understand what is actually meant by the above phrase.)\r\n\r\n  * (I don't think the phrase is meant to denote the same thing as `::std::sys::os::page_size`; so I don't think that would be a way to query the actual value, though certainly it would provide a *bound* that a programmer can use in the meantime...)\r\n  * IMO, if we were to add a method to query the value of \"largest supported alignment\", it should be part of the low-level allocator interface (see [RFC 1183][]), since presumably the largest supported value would be connected to the allocator implementation.\r\n\r\n[RFC 1183]: https://github.com/rust-lang/rfcs/blob/master/text/1183-swap-out-jemalloc.md\r\n\r\n#### Post-validation is not absurd\r\n\r\nGiven that at least *some* alloc-crate + target combinations are not imposing the above requirement (in the sense that they return null when the given alignment is unsatisfiable), it seems somewhat reasonable to me to add the post-allocation check that I described above, as long as we do it in a conditionalized manner.\r\n\r\n  * Instead of conditionalizing based on the target OS, we might be able to write the injected code in a way where the compiler can optimize away the check when the given alignment is known and falls beneath some reasonable bound, like `align <= 0x1000`, working under the assumptions that all allocators will behave properly with such an input.\r\n  * (I suspect in a vast number of cases, the alignment is statically known, so this would probably be a fine solution.)\r\n\r\n\r\nAlso, the man page for `posix_memalign` on my version of OS X says nothing about an upper bound on the value for `alignment`. This to me indicates that this is a *bug* in OS X itself, which we can report and workaround via post-validation in the short term. \r\n\r\n *  (If we did conditionalize based on target OS, then longer term, I don't know how we would best deal with trying to conditionally inject the post-validation in question; perhaps the default `#[allocator]` crate linking could choose between the two variants depending on which version of OS X is targetted.)\r\n\r\n### The Sample Program\r\n\r\nHere is that sample program (you can toggle the different allocates with `--cfg alloc_system` or `--cfg alloc_jemalloc`).  When you run it, the thing to look for is when there is an output line that has `rem: VALUE` on the end, which is only printed for non-zero remainder (when one divides the address by the alignment).\r\n\r\n(Also, here's a [playpen](https://play.rust-lang.org/?gist=6fa7a76ab99380251920&version=nightly), though I repeat that the problematic behavior does not exhibit itself on versions of Linux that I have tried.)\r\n\r\n```rust\r\n#![feature(alloc, heap_api)]                                                                                                \r\n#![cfg_attr(alloc_jemalloc, feature(alloc_jemalloc))]                                                                       \r\n#![cfg_attr(alloc_system, feature(alloc_system))]                                                                           \r\nextern crate alloc;                                                                                                         \r\n#[cfg(alloc_jemalloc)]                                                                                                      \r\nextern crate alloc_jemalloc;                                                                                                \r\n#[cfg(alloc_system)]                                                                                                        \r\nextern crate alloc_system;                                                                                                  \r\nuse std::{isize, mem, ptr};                                                                                                 \r\nconst PRINT_ALL: bool = false;                                                                                              \r\nfn main() {                                                                                                                 \r\n    unsafe {                                                                                                                \r\n        for i in 0 .. mem::size_of::<usize>()*8 {                                                                           \r\n            let (mut v, a) = (Vec::new(), 1 << i);                                                                          \r\n            let try_alloc = |j, s, a| {                                                                                     \r\n                let p = alloc::heap::allocate(s, a);                                                                        \r\n                let q = p as usize;                                                                                         \r\n                let rem = q % a;                                                                                            \r\n                if p == ptr::null_mut() {                                                                                   \r\n                    println!(\"(i,j,s):{ijs:>30} a:{a:>8}  =>  p:{p:20?}\",                                                   \r\n                             ijs=format!(\"({},{},0x{:x})\", i,j,s),                                                          \r\n                             a=format!(\"1<<{}\", i),                                                                         \r\n                             p=p);                                                                                          \r\n                } else if PRINT_ALL || rem != 0 || p == ptr::null_mut() {                                                   \r\n                    println!(\"(i,j,s):{ijs:>30} a:{a:>8}  =>  p:{p:20?}, rem: 0x{rem:x}\",                                   \r\n                             ijs=format!(\"({},{},0x{:x})\", i,j,s),                                                          \r\n                             a=format!(\"1<<{}\", i),                                                                         \r\n                             p=p,                                                                                           \r\n                             rem=rem);                                                                                      \r\n                }                                                                                                           \r\n                p                                                                                                           \r\n            };                                                                                                              \r\n            {                                                                                                               \r\n                let mut alloc_loop = |init_s, start, limit| {                                                               \r\n                    let mut s = init_s;                                                                                     \r\n                    for j in start .. limit {                                                                               \r\n                        if s > isize::MAX as usize {                                                                        \r\n                            println!(\"breaking b/c s={} > isize::MAX={}\", s, isize::MAX);                                   \r\n                            break;                                                                                          \r\n                        }                                                                                                   \r\n                        let p = try_alloc(j, s, a);                                                                         \r\n                        if p != ptr::null_mut() { v.push((p,s,a)); } else { break; }                                        \r\n                        s += j;                                                                                             \r\n                    }                                                                                                       \r\n                };                                                                                                          \r\n\r\n                if i >= 8*mem::size_of::<usize>() { break; }                                                                \r\n                alloc_loop(10, 0, 10);                                                                                      \r\n                alloc_loop(a, 10, 20);                                                                                      \r\n            }                                                                                                               \r\n\r\n            for (p,s,a) in v { alloc::heap::deallocate(p, s, a); }                                                          \r\n        }                                                                                                                   \r\n    }                                                                                                                       \r\n}                                                                                                                           \r\n```\r\n\r\n#### Output on Mac OS X showing the erroneous value(s)\r\n\r\nSee following gist: https://gist.github.com/pnkfelix/535e8d4e810025331c77"}}