{"action":"created","issue":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/1088","labels_url":"https://api.github.com/repos/open-mpi/ompi/issues/1088/labels{/name}","comments_url":"https://api.github.com/repos/open-mpi/ompi/issues/1088/comments","events_url":"https://api.github.com/repos/open-mpi/ompi/issues/1088/events","html_url":"https://github.com/open-mpi/ompi/issues/1088","id":114628985,"number":1088,"title":"pmix1xx segfault when orte-dvm put under load","user":{"login":"marksantcroos","id":383387,"avatar_url":"https://avatars.githubusercontent.com/u/383387?v=3","gravatar_id":"","url":"https://api.github.com/users/marksantcroos","html_url":"https://github.com/marksantcroos","followers_url":"https://api.github.com/users/marksantcroos/followers","following_url":"https://api.github.com/users/marksantcroos/following{/other_user}","gists_url":"https://api.github.com/users/marksantcroos/gists{/gist_id}","starred_url":"https://api.github.com/users/marksantcroos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marksantcroos/subscriptions","organizations_url":"https://api.github.com/users/marksantcroos/orgs","repos_url":"https://api.github.com/users/marksantcroos/repos","events_url":"https://api.github.com/users/marksantcroos/events{/privacy}","received_events_url":"https://api.github.com/users/marksantcroos/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"rhc54","id":2440928,"avatar_url":"https://avatars.githubusercontent.com/u/2440928?v=3","gravatar_id":"","url":"https://api.github.com/users/rhc54","html_url":"https://github.com/rhc54","followers_url":"https://api.github.com/users/rhc54/followers","following_url":"https://api.github.com/users/rhc54/following{/other_user}","gists_url":"https://api.github.com/users/rhc54/gists{/gist_id}","starred_url":"https://api.github.com/users/rhc54/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhc54/subscriptions","organizations_url":"https://api.github.com/users/rhc54/orgs","repos_url":"https://api.github.com/users/rhc54/repos","events_url":"https://api.github.com/users/rhc54/events{/privacy}","received_events_url":"https://api.github.com/users/rhc54/received_events","type":"User","site_admin":false},"milestone":null,"comments":8,"created_at":"2015-11-02T16:19:10Z","updated_at":"2015-11-04T21:38:36Z","closed_at":null,"body":"While hammering with orte-submit on a 10x24 core orte-dvm I repeatedly run into the segfault below.\r\nThe workload is a set of single core tasks equal the size of the number of cores (240 in this case).\r\nThe duration of the tasks doesn't influence it, as it happens at startup.\r\n\r\nThis output is captured with a build with `--enable-mem-debug` enabled.\r\n\r\n```\r\n$ orte-dvm --report-uri dvm_uri \r\nDVM ready\r\nmalloc debug: Realloc 0x7f9abfff7eb0 for 36864 bytes failed (../../../../../../opal/mca/pmix/pmix1xx/pmix/src/buffer_ops/internal_functions.c, 65)\r\n[comet-ln3:28029] *** Process received signal ***\r\n[comet-ln3:28029] Signal: Segmentation fault (11)\r\n[comet-ln3:28029] Signal code: Address not mapped (1)\r\n[comet-ln3:28029] Failing at address: 0x4a\r\n[comet-ln3:28029] [ 0] /lib64/libpthread.so.0[0x331f60f710]\r\n[comet-ln3:28029] [ 1] /lib64/libc.so.6[0x331ee83da3]\r\n[comet-ln3:28029] [ 2] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_buffer_extend+0x176)[0x7f9b0810b2e4]\r\n[comet-ln3:28029] [ 3] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_pack_byte+0x55)[0x7f9b0810e0d6]\r\n[comet-ln3:28029] [ 4] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_pack_string+0xfb)[0x7f9b0810e552]\r\n[comet-ln3:28029] [ 5] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_pack_buffer+0xc7)[0x7f9b0810de9e]\r\n[comet-ln3:28029] [ 6] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(+0x1390d)[0x7f9b0810e90d]\r\n[comet-ln3:28029] [ 7] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_pack_value+0x9a)[0x7f9b0810ed51]\r\n[comet-ln3:28029] [ 8] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_pack_kval+0xb4)[0x7f9b0810f585]\r\n[comet-ln3:28029] [ 9] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_pack_buffer+0xc7)[0x7f9b0810de9e]\r\n[comet-ln3:28029] [10] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_pack+0x8d)[0x7f9b0810ddd5]\r\n[comet-ln3:28029] [11] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(+0x382ea)[0x7f9b081332ea]\r\n[comet-ln3:28029] [12] /home/marksant/openmpi/installed/STATIC/lib/libopen-pal.so.0(opal_libevent2022_event_base_loop+0xd81)[0x7f9b2d7b5041]\r\n[comet-ln3:28029] [13] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(+0x218dd)[0x7f9b0811c8dd]\r\n[comet-ln3:28029] [14] /lib64/libpthread.so.0[0x331f6079d1]\r\n[comet-ln3:28029] [15] /lib64/libc.so.6(clone+0x6d)[0x331eee88fd]\r\n[comet-ln3:28029] *** End of error message ***\r\nSegmentation fault\r\n```\r\n\r\nAt the side of the orte-submit client I get errors like:\r\n```\r\n[comet-ln3:30834] *** Process received signal ***\r\n[comet-ln3:30834] Signal: Segmentation fault (11)\r\n[comet-ln3:30834] Signal code: Address not mapped (1)\r\n[comet-ln3:30834] Failing at address: (nil)\r\n[comet-ln3:30834] [ 0] /lib64/libpthread.so.0[0x331f60f710]\r\n[comet-ln3:30834] *** End of error message ***\r\n```\r\n\r\nAny ideas and or suggestions to further debug this?"},"comment":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/comments/153874475","html_url":"https://github.com/open-mpi/ompi/issues/1088#issuecomment-153874475","issue_url":"https://api.github.com/repos/open-mpi/ompi/issues/1088","id":153874475,"user":{"login":"marksantcroos","id":383387,"avatar_url":"https://avatars.githubusercontent.com/u/383387?v=3","gravatar_id":"","url":"https://api.github.com/users/marksantcroos","html_url":"https://github.com/marksantcroos","followers_url":"https://api.github.com/users/marksantcroos/followers","following_url":"https://api.github.com/users/marksantcroos/following{/other_user}","gists_url":"https://api.github.com/users/marksantcroos/gists{/gist_id}","starred_url":"https://api.github.com/users/marksantcroos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marksantcroos/subscriptions","organizations_url":"https://api.github.com/users/marksantcroos/orgs","repos_url":"https://api.github.com/users/marksantcroos/repos","events_url":"https://api.github.com/users/marksantcroos/events{/privacy}","received_events_url":"https://api.github.com/users/marksantcroos/received_events","type":"User","site_admin":false},"created_at":"2015-11-04T21:38:36Z","updated_at":"2015-11-04T21:38:36Z","body":"Here another one.\r\n\r\nNow with only 1 node of 24 cores.\r\nStarting 24 `orte-submits -np 1` in \"parallel\" with the script above in a loop.\r\n\r\nIt feels like a classical race condition, as the smaller the problem size, the longer it takes to reproduce.\r\n\r\n```\r\nmalloc debug: Realloc 0x7fac6bff7eb0 for 36864 bytes failed (../../../../../../opal/mca/pmix/pmix1xx/pmix/src/buffer_ops/internal_functions.c, 65)\r\n[comet-ln3:28482] *** Process received signal ***\r\n[comet-ln3:28482] Signal: Segmentation fault (11)\r\n[comet-ln3:28482] Signal code: Address not mapped (1)\r\n[comet-ln3:28482] Failing at address: 0x4a\r\n[comet-ln3:28482] [ 0] /lib64/libpthread.so.0[0x331f60f710]\r\n[comet-ln3:28482] [ 1] /lib64/libc.so.6[0x331ee83da3]\r\n[comet-ln3:28482] [ 2] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_buffer_extend+0x176)[0x7facb42332e4]\r\n[comet-ln3:28482] [ 3] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_pack_byte+0x55)[0x7facb42360d6]\r\n[comet-ln3:28482] [ 4] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_pack_string+0xfb)[0x7facb4236552]\r\n[comet-ln3:28482] [ 5] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_pack_buffer+0xc7)[0x7facb4235e9e]\r\n[comet-ln3:28482] [ 6] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(+0x1390d)[0x7facb423690d]\r\n[comet-ln3:28482] [ 7] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_pack_value+0x9a)[0x7facb4236d51]\r\n[comet-ln3:28482] [ 8] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_pack_kval+0xb4)[0x7facb4237585]\r\n[comet-ln3:28482] [ 9] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_pack_buffer+0xc7)[0x7facb4235e9e]\r\n[comet-ln3:28482] [10] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(opal_pmix_pmix1xx_pmix_bfrop_pack+0x8d)[0x7facb4235dd5]\r\n[comet-ln3:28482] [11] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(+0x382ea)[0x7facb425b2ea]\r\n[comet-ln3:28482] [12] /home/marksant/openmpi/installed/STATIC/lib/libopen-pal.so.0(opal_libevent2022_event_base_loop+0xd81)[0x7facd98dd041]\r\n[comet-ln3:28482] [13] /home/marksant/openmpi/installed/STATIC/lib/libpmix.so.0(+0x218dd)[0x7facb42448dd]\r\n[comet-ln3:28482] [14] /lib64/libpthread.so.0[0x331f6079d1]\r\n[comet-ln3:28482] [15] /lib64/libc.so.6(clone+0x6d)[0x331eee88fd]\r\n[comet-ln3:28482] *** End of error message ***\r\nSegmentation fault\r\n```"}}