{"action":"closed","issue":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/1145","labels_url":"https://api.github.com/repos/open-mpi/ompi/issues/1145/labels{/name}","comments_url":"https://api.github.com/repos/open-mpi/ompi/issues/1145/comments","events_url":"https://api.github.com/repos/open-mpi/ompi/issues/1145/events","html_url":"https://github.com/open-mpi/ompi/issues/1145","id":118336038,"number":1145,"title":"Race condition with ring_c.c","user":{"login":"artpol84","id":7534910,"avatar_url":"https://avatars.githubusercontent.com/u/7534910?v=3","gravatar_id":"","url":"https://api.github.com/users/artpol84","html_url":"https://github.com/artpol84","followers_url":"https://api.github.com/users/artpol84/followers","following_url":"https://api.github.com/users/artpol84/following{/other_user}","gists_url":"https://api.github.com/users/artpol84/gists{/gist_id}","starred_url":"https://api.github.com/users/artpol84/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/artpol84/subscriptions","organizations_url":"https://api.github.com/users/artpol84/orgs","repos_url":"https://api.github.com/users/artpol84/repos","events_url":"https://api.github.com/users/artpol84/events{/privacy}","received_events_url":"https://api.github.com/users/artpol84/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/open-mpi/ompi/labels/blocker","name":"blocker","color":"e11d21"},{"url":"https://api.github.com/repos/open-mpi/ompi/labels/bug","name":"bug","color":"fc2929"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/open-mpi/ompi/milestones/4","html_url":"https://github.com/open-mpi/ompi/milestones/v2.0.0","labels_url":"https://api.github.com/repos/open-mpi/ompi/milestones/4/labels","id":810131,"number":4,"title":"v2.0.0","description":"","creator":{"login":"ompiteam","id":2165735,"avatar_url":"https://avatars.githubusercontent.com/u/2165735?v=3","gravatar_id":"","url":"https://api.github.com/users/ompiteam","html_url":"https://github.com/ompiteam","followers_url":"https://api.github.com/users/ompiteam/followers","following_url":"https://api.github.com/users/ompiteam/following{/other_user}","gists_url":"https://api.github.com/users/ompiteam/gists{/gist_id}","starred_url":"https://api.github.com/users/ompiteam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ompiteam/subscriptions","organizations_url":"https://api.github.com/users/ompiteam/orgs","repos_url":"https://api.github.com/users/ompiteam/repos","events_url":"https://api.github.com/users/ompiteam/events{/privacy}","received_events_url":"https://api.github.com/users/ompiteam/received_events","type":"User","site_admin":false},"open_issues":25,"closed_issues":181,"state":"open","created_at":"2014-10-01T15:58:55Z","updated_at":"2015-12-15T12:33:08Z","due_on":null,"closed_at":null},"comments":8,"created_at":"2015-11-23T08:15:48Z","updated_at":"2015-12-15T12:33:08Z","closed_at":"2015-12-15T12:33:08Z","body":"@hjelmn @rhc54  @miked-mellanox @jladd-mlnx (please, include others who may be involved)\r\nHello,\r\nI'm seeing rear hangs (one for several hundreds of runs) of ring_c.c application from examples when running at medium scale (1792 - 3584 processes).\r\nI'm using SLURM direct launch (testing PMIx component). This problem appears both with PMI2 and PMIx SLURM plugins so I don't think that it is related to this component.\r\n\r\nI'm using latest master:\r\n```\r\ncommit a549db8ce20708fcc991f72a96f827aabb27cf99\r\nAuthor: Jeff Squyres <jsquyres@cisco.com>\r\nDate:   Fri Nov 20 22:42:01 2015 -0500\r\ncreate_tarball.sh: use \"autogen.pl --force\"\r\n```\r\nEnvironment is following:\r\n```\r\nexport OMPI_MCA_mpi_add_procs_cutoff=10000\r\nexport OMPI_MCA_pml=ob1\r\nexport OMPI_MCA_btl=\"self,vader,openib\"\r\n```\r\n\r\nUsing binary search I was able to find processes that fail to exchange with messages:\r\n\r\n**node *x*, last rank on the node:**\r\n```\r\n(gdb) bt\r\n#0  0x0000003b808cf247 in sched_yield () from /lib64/libc.so.6\r\n#1  0x00007ffff765cb79 in opal_progress () at runtime/opal_progress.c:199\r\n#2  0x00007ffff3cf292a in opal_condition_wait (c=0x7ffff7ffab20, m=0x7ffff7ffaaa0) at ../../../../opal/threads/condition.h:76\r\n#3  0x00007ffff3cf2b0e in ompi_request_wait_completion (req=0xf95600) at ../../../../ompi/request/request.h:383\r\n#4  0x00007ffff3cf4b7b in mca_pml_ob1_send (buf=0x7fffffffd138, count=1, datatype=0x601260, dst=2044, tag=201,\r\n    sendmode=MCA_PML_BASE_SEND_STANDARD, comm=0x601460) at pml_ob1_isend.c:251\r\n#5  0x00007ffff7d1dac6 in PMPI_Send (buf=0x7fffffffd138, count=1, type=0x601260, dest=2044, tag=201, comm=0x601460) at psend.c:78\r\n#6  0x00000000004009b9 in main (argc=1, argv=0x7fffffffd248) at ring_c.c:60\r\n(gdb) frame 6\r\n#6  0x00000000004009b9 in main (argc=1, argv=0x7fffffffd248) at ring_c.c:60\r\n60              MPI_Send(&message, 1, MPI_INT, next, tag, MPI_COMM_WORLD);\r\n(gdb) p message\r\n$1 = 10\r\n(gdb) l\r\n55              if (0 == rank) {\r\n56                  --message;\r\n57                  printf(\"Process 0 decremented value: %d\\n\", message);\r\n58              }\r\n59\r\n60              MPI_Send(&message, 1, MPI_INT, next, tag, MPI_COMM_WORLD);\r\n61              if (0 == message) {\r\n62                  printf(\"Process %d exiting\\n\", rank);\r\n63                  break;\r\n64              }\r\n(gdb) p rank\r\n$2 = 2043\r\n```\r\n\r\n**node (*x*+1), first rank on the node:**\r\n```\r\n(gdb) bt\r\n#0  0x00000034fb6cf247 in sched_yield () from /lib64/libc.so.6\r\n#1  0x00007ffff765cb79 in opal_progress () at runtime/opal_progress.c:199\r\n#2  0x00007ffff3cf09a9 in opal_condition_wait (c=0x7ffff7ffab20, m=0x7ffff7ffaaa0) at ../../../../opal/threads/condition.h:76\r\n#3  0x00007ffff3cf0b70 in ompi_request_wait_completion (req=0xfab6c0) at ../../../../ompi/request/request.h:383\r\n#4  0x00007ffff3cf1d05 in mca_pml_ob1_recv (addr=0x7fffffffd118, count=1, datatype=0x601260, src=2043, tag=201, comm=0x601460, status=0x0)\r\n    at pml_ob1_irecv.c:123\r\n#5  0x00007ffff7d146b2 in PMPI_Recv (buf=0x7fffffffd118, count=1, type=0x601260, source=2043, tag=201, comm=0x601460, status=0x0)\r\n    at precv.c:79\r\n#6  0x000000000040096b in main (argc=1, argv=0x7fffffffd228) at ring_c.c:52\r\n(gdb) frame 6\r\n#6  0x000000000040096b in main (argc=1, argv=0x7fffffffd228) at ring_c.c:52\r\n52              MPI_Recv(&message, 1, MPI_INT, prev, tag, MPI_COMM_WORLD,\r\n(gdb) p message\r\n$2 = 4196928\r\n(gdb) l\r\n47             passes the message on to the next process and then quits.  By\r\n48             passing the 0 message first, every process gets the 0 message\r\n49             and can quit normally. */\r\n50\r\n51          while (1) {\r\n52              MPI_Recv(&message, 1, MPI_INT, prev, tag, MPI_COMM_WORLD,\r\n53                       MPI_STATUS_IGNORE);\r\n54\r\n55              if (0 == rank) {\r\n56                  --message;\r\n(gdb) p rank\r\n$3 = 2044\r\n```\r\n*x* may vary according to my observations."}}