{"action":"created","issue":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/638","labels_url":"https://api.github.com/repos/open-mpi/ompi/issues/638/labels{/name}","comments_url":"https://api.github.com/repos/open-mpi/ompi/issues/638/comments","events_url":"https://api.github.com/repos/open-mpi/ompi/issues/638/events","html_url":"https://github.com/open-mpi/ompi/issues/638","id":87393742,"number":638,"title":"Problem with the default value of btl_openib_memalign_threshold","user":{"login":"larrystevenwise","id":9438087,"avatar_url":"https://avatars.githubusercontent.com/u/9438087?v=3","gravatar_id":"","url":"https://api.github.com/users/larrystevenwise","html_url":"https://github.com/larrystevenwise","followers_url":"https://api.github.com/users/larrystevenwise/followers","following_url":"https://api.github.com/users/larrystevenwise/following{/other_user}","gists_url":"https://api.github.com/users/larrystevenwise/gists{/gist_id}","starred_url":"https://api.github.com/users/larrystevenwise/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/larrystevenwise/subscriptions","organizations_url":"https://api.github.com/users/larrystevenwise/orgs","repos_url":"https://api.github.com/users/larrystevenwise/repos","events_url":"https://api.github.com/users/larrystevenwise/events{/privacy}","received_events_url":"https://api.github.com/users/larrystevenwise/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"larrystevenwise","id":9438087,"avatar_url":"https://avatars.githubusercontent.com/u/9438087?v=3","gravatar_id":"","url":"https://api.github.com/users/larrystevenwise","html_url":"https://github.com/larrystevenwise","followers_url":"https://api.github.com/users/larrystevenwise/followers","following_url":"https://api.github.com/users/larrystevenwise/following{/other_user}","gists_url":"https://api.github.com/users/larrystevenwise/gists{/gist_id}","starred_url":"https://api.github.com/users/larrystevenwise/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/larrystevenwise/subscriptions","organizations_url":"https://api.github.com/users/larrystevenwise/orgs","repos_url":"https://api.github.com/users/larrystevenwise/repos","events_url":"https://api.github.com/users/larrystevenwise/events{/privacy}","received_events_url":"https://api.github.com/users/larrystevenwise/received_events","type":"User","site_admin":false},"milestone":null,"comments":13,"created_at":"2015-06-11T15:56:12Z","updated_at":"2015-06-12T16:49:25Z","closed_at":null,"body":"\r\n\r\nEmail thread:\r\n\r\n\r\nRalph --\r\n\r\nThis change was not correct (https://github.com/open-mpi/ompi/commit/ce915b5757d428d3e914dcef50bd4b2636561bca).  It is causing memory corruption in the openib BTL.\r\n\r\n\r\n\r\n> On May 25, 2015, at 11:56 AM, Ralph Castain <rhc@open-mpi.org> wrote:\r\n>\r\n> I don’t see a problem with it. FWIW: I’m getting ready to release 1.8.6 in the next week\r\n>\r\n>\r\n>> On May 25, 2015, at 8:46 AM, Xavier Besseron <xavier.besseron@uni.lu> wrote:\r\n>>\r\n>> Good that it will be fixed in the next release!\r\n>>\r\n>> In the meantime, and because it might impact other users,\r\n>> I would like to ask my sysadmins to set btl_openib_memalign_threshold=12288 in etc/openmpi-mca-params.conf on our clusters.\r\n>>\r\n>> Do you see any good reason not doing it?\r\n>>\r\n>> Thanks!\r\n>>\r\n>>\r\n>> Xavier\r\n>>\r\n>>\r\n>>\r\n>> On Mon, May 25, 2015 at 4:12 PM, Ralph Castain <rhc@open-mpi.org> wrote:\r\n>> I found the problem - someone had a typo in btl_openib_mca.c. The threshold need to be set to the module eager limit as that is the only thing defined at that point.\r\n>>\r\n>> Thanks for bringing it to our attention! I’ll set it up to go into 1.8.6\r\n>>\r\n>>\r\n>>> On May 25, 2015, at 3:04 AM, Xavier Besseron <xavier.besseron@uni.lu> wrote:\r\n>>>\r\n>>> Hi,\r\n>>>\r\n>>> Thanks for your reply Ralph.\r\n>>>\r\n>>> The option only I'm using when configuring OpenMPI is '--prefix'.\r\n>>> When checking the config.log file, I see \r\n>>>\r\n>>> configure:208504: checking whether the openib BTL will use malloc hooks\r\n>>> configure:208510: result: yes\r\n>>>\r\n>>> so I guess it is properly enabled (full config.log in attachment of this email).\r\n>>>\r\n>>>\r\n>>>\r\n>>> However, I think I have the reason of the bug (lines refer to source code of OpenMPI 1.8.5):\r\n>>>\r\n>>> The default value of memalign_threshold is taken from eager_limit in function btl_openib_register_mca_params() in btl_openib_mca.c line 717.\r\n>>> But the default value is eager_limit is set in btl_openib_component.c at line 193 right after the call to btl_openib_register_mca_params().\r\n>>>\r\n>>> To summarize, memalign_threshold gets its value from eager_limit before this one gets its value assigned.\r\n>>>\r\n>>>\r\n>>>\r\n>>> Best regards,\r\n>>>\r\n>>> Xavier\r\n>>>\r\n>>>\r\n>>>\r\n>>>  \r\n>>>\r\n>>>\r\n>>>\r\n>>>\r\n>>> On Mon, May 25, 2015 at 2:27 AM, Ralph Castain <rhc@open-mpi.org> wrote:\r\n>>> Looking at the code, we do in fact set the memalign_threshold = eager_limit by default, but only if you configured with —enable-btl-openib-malloc-alignment AND/OR we found the malloc hook functions were available.\r\n>>>\r\n>>> You might check config.log to see if the openib malloc hooks were enabled. My guess is that they weren’t, for some reason.\r\n>>>\r\n>>>\r\n>>>> On May 24, 2015, at 9:07 AM, Xavier Besseron <xavier.besseron@uni.lu> wrote:\r\n>>>>\r\n>>>> Dear OpenMPI developers / users,\r\n>>>>\r\n>>>> This is much more a comment than a question since I believe I have already solved my issue. But I would like to report it.\r\n>>>>\r\n>>>> I have noticed my code performed very badly with OpenMPI when Infinand is enabled, sometime +50% or even +100% overhead.\r\n>>>> I also have this slowdown when running with one thread and one process. In such case, there is no other MPI call than MPI_Init() and MPI_Finalize().\r\n>>>> This overhead disappears if I disable at runtime the openib btl, ie with '--mca btl ^openib'.\r\n>>>> After further investigation, I figured out it comes from the memory allocator which is aligning every memory allocation when Infiniband is used.\r\n>>>> This makes sense because my code is a large irregular C++ code creating and deleting many objects.\r\n>>>>\r\n>>>> Just below is the documentation of the relevant MCA parameters coming ompi_info:\r\n>>>>\r\n>>>> MCA btl: parameter \"btl_openib_memalign\" (current value: \"32\", data source: default, level: 9 dev/all, type: int)\r\n>>>>          [64 | 32 | 0] - Enable (64bit or 32bit)/Disable(0) memoryalignment for all malloc calls if btl openib is used.\r\n>>>>\r\n>>>> MCA btl: parameter \"btl_openib_memalign_threshold\" (current value: \"0\", data source: default, level: 9 dev/all, type: size_t)\r\n>>>>          Allocating memory more than btl_openib_memalign_threshholdbytes will automatically be algined to the value of btl_openib_memalign bytes.memalign_threshhold defaults to the same value as mca_btl_openib_eager_limit.\r\n>>>>\r\n>>>> MCA btl: parameter \"btl_openib_eager_limit\" (current value: \"12288\", data source: default, level: 4 tuner/basic, type: size_t)\r\n>>>>          Maximum size (in bytes, including header) of \"short\" messages (must be >= 1).\r\n>>>>\r\n>>>>\r\n>>>> In the end, the problem is that the default value for btl_openib_memalign_threshold is 0, which means that all memory allocations are aligned to 32 bits.\r\n>>>> The documentation says that the default value of btl_openib_memalign_threshold should be the the same as btl_openib_eager_limit, ie 12288 instead of 0.\r\n>>>>\r\n>>>> On my side, changing btl_openib_memalign_threshold to 12288 fixes my performance issue.\r\n>>>> However, I believe that the default value of btl_openib_memalign_threshold should be fixed in the OpenMPI code (or at least the documentation should be fixed).\r\n>>>>\r\n>>>> I tried OpenMPI 1.8.5, 1.7.3 and 1.6.4 and it's all the same.\r\n>>>>\r\n>>>>\r\n>>>> Bonus question:\r\n>>>> As this issue might impact other users, I'm considering applying a global fix on our clusters by setting this default value etc/openmpi-mca-params.conf.\r\n>>>> Do you see any good reason not doing it?\r\n>>>>\r\n>>>> Thank you for your comments.\r\n>>>>\r\n>>>> Best regards,\r\n>>>>\r\n>>>> Xavier\r\n>>>>\r\n>>>>\r\n>>>> -- \r\n>>>> Dr Xavier BESSERON\r\n>>>> Research associate\r\n>>>> FSTC, University of Luxembourg\r\n>>>> Campus Kirchberg, Office E-007\r\n>>>> Phone: +352 46 66 44 5418\r\n>>>> http://luxdem.uni.lu/"},"comment":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/comments/111554071","html_url":"https://github.com/open-mpi/ompi/issues/638#issuecomment-111554071","issue_url":"https://api.github.com/repos/open-mpi/ompi/issues/638","id":111554071,"user":{"login":"larrystevenwise","id":9438087,"avatar_url":"https://avatars.githubusercontent.com/u/9438087?v=3","gravatar_id":"","url":"https://api.github.com/users/larrystevenwise","html_url":"https://github.com/larrystevenwise","followers_url":"https://api.github.com/users/larrystevenwise/followers","following_url":"https://api.github.com/users/larrystevenwise/following{/other_user}","gists_url":"https://api.github.com/users/larrystevenwise/gists{/gist_id}","starred_url":"https://api.github.com/users/larrystevenwise/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/larrystevenwise/subscriptions","organizations_url":"https://api.github.com/users/larrystevenwise/orgs","repos_url":"https://api.github.com/users/larrystevenwise/repos","events_url":"https://api.github.com/users/larrystevenwise/events{/privacy}","received_events_url":"https://api.github.com/users/larrystevenwise/received_events","type":"User","site_admin":false},"created_at":"2015-06-12T16:49:25Z","updated_at":"2015-06-12T16:49:25Z","body":"I cannot reproduce it on top-of-tree from git, nor on 1.8.5.  @jsquyres, can you please retry your experiments, maybe on a 2nd system?"}}