{"action":"created","issue":{"url":"https://api.github.com/repos/flux-framework/flux-core/issues/411","labels_url":"https://api.github.com/repos/flux-framework/flux-core/issues/411/labels{/name}","comments_url":"https://api.github.com/repos/flux-framework/flux-core/issues/411/comments","events_url":"https://api.github.com/repos/flux-framework/flux-core/issues/411/events","html_url":"https://github.com/flux-framework/flux-core/issues/411","id":107105290,"number":411,"title":"Specifying only n-nodes without n-tasks causes wreck/pmi failures, PMGR occasionally pretends to work","user":{"login":"trws","id":660149,"avatar_url":"https://avatars.githubusercontent.com/u/660149?v=3","gravatar_id":"","url":"https://api.github.com/users/trws","html_url":"https://github.com/trws","followers_url":"https://api.github.com/users/trws/followers","following_url":"https://api.github.com/users/trws/following{/other_user}","gists_url":"https://api.github.com/users/trws/gists{/gist_id}","starred_url":"https://api.github.com/users/trws/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/trws/subscriptions","organizations_url":"https://api.github.com/users/trws/orgs","repos_url":"https://api.github.com/users/trws/repos","events_url":"https://api.github.com/users/trws/events{/privacy}","received_events_url":"https://api.github.com/users/trws/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-09-18T00:09:22Z","updated_at":"2015-09-18T20:15:41Z","closed_at":null,"body":"Having gotten some of the kinks worked out of profiling a broker, I went to run some MPI tests with it, and seem not to be able to run anything MPI-based on more than one node.\r\n\r\nOn a single node, mpi_hello works fine, though 3 seconds seems... unfortuante... for just four processes:\r\n```\r\nflux wreckrun -n 4 ./mpi_hello\r\nlt-flux-broker: [1442534481.147908] job.info[0] got request job.create\r\nlt-flux-broker: [1442534481.149268] job.info[0] Setting job 6 to reserved\r\nlt-flux-broker: [1442534481.191392] lwj.6.debug[0] initializing from CMB: rank=0\r\nlt-flux-broker: [1442534481.192444] lwj.6.debug[0] lwj.6: node0: basis=0\r\nlt-flux-broker: [1442534481.196138] lwj.6.info[0] lwj 6: node0: nprocs=4, nnodes=1, cmdline=[ \".\\/mpi_hello\" ]\r\nlt-flux-broker: [1442534481.196205] lwj.6.debug[0] updating job state to starting\r\nlt-flux-broker: [1442534481.197697] lwj.6.debug[0] reading lua files from /g/g12/scogland/projects/flux/flux-core/src/modules/wreck/lua.d/*.lua\r\nlt-flux-broker: [1442534481.202228] lwj.6.debug[0] task0: pid 242528 (./mpi_hello): started\r\nlt-flux-broker: [1442534481.202427] lwj.6.debug[0] task1: pid 242529 (./mpi_hello): started\r\nlt-flux-broker: [1442534481.202634] lwj.6.debug[0] task2: pid 242530 (./mpi_hello): started\r\nlt-flux-broker: [1442534481.202831] lwj.6.debug[0] task3: pid 242531 (./mpi_hello): started\r\nlt-flux-broker: [1442534481.204447] lwj.6.debug[0] updating job state to running\r\n0: completed MPI_Init in 2.855s.  There are 4 tasks\r\nlt-flux-broker: [1442534484.215292] lwj.6.debug[0] task0: pid 242528 (./mpi_hello) exited with status 0x0000\r\nlt-flux-broker: [1442534484.215803] lwj.6.debug[0] task1: pid 242529 (./mpi_hello) exited with status 0x0000\r\nlt-flux-broker: [1442534484.216323] lwj.6.debug[0] task2: pid 242530 (./mpi_hello) exited with status 0x0000\r\nlt-flux-broker: [1442534484.216742] lwj.6.debug[0] task3: pid 242531 (./mpi_hello) exited with status 0x0000\r\nlt-flux-broker: [1442534484.218629] lwj.6.debug[0] updating job state to complete\r\nlt-flux-broker: [1442534484.220869] lwj.6.info[0] job complete. exiting...\r\n0: completed first barrier\r\n0: completed MPI_Finalize\r\nlt-flux-broker: [1442534484.223804] job.info[0] got request job.disconnect\r\n```\r\n\r\nRunning it on four nodes, one process per node, initially gave me one task on one node and PMGR collective errors on the other three, like this:\r\n```\r\nflux wreckrun -N 4 ./mpi_hello\r\nlt-flux-broker: [1442534264.775650] job.info[0] got request job.create\r\nlt-flux-broker: [1442534264.776906] job.info[0] Setting job 4 to reserved\r\nlt-flux-broker: waitpid: No child processes\r\nlt-flux-broker: [1442534264.810765] lwj.4.debug[0] initializing from CMB: rank=0\r\nlt-flux-broker: [1442534264.811804] lwj.4.debug[0] lwj.4: node0: basis=0\r\nlt-flux-broker: [1442534264.813179] lwj.4.info[0] lwj 4: node0: nprocs=1, nnodes=4, cmdline=[ \".\\/mpi_hello\" ]\r\nlt-flux-broker: [1442534264.813249] lwj.4.debug[0] updating job state to starting\r\nlt-flux-broker: [1442534264.814502] lwj.4.debug[0] reading lua files from /g/g12/scogland/projects/flux/flux-core/src/modules/wreck/lua.d/*.lua\r\nlt-flux-broker: [1442534264.818279] lwj.4.debug[0] task0: pid 242448 (./mpi_hello): started\r\nlt-flux-broker: [1442534264.832277] lwj.4.debug[1] initializing from CMB: rank=1\r\nlt-flux-broker: [1442534264.832419] lwj.4.debug[2] initializing from CMB: rank=2\r\nlt-flux-broker: [1442534264.832824] lwj.4.debug[1] lwj.4: node1: basis=1\r\nlt-flux-broker: [1442534264.832659] lwj.4.debug[3] initializing from CMB: rank=3\r\nlt-flux-broker: [1442534264.833404] lwj.4.info[1] lwj 4: node1: nprocs=1, nnodes=4, cmdline=[ \".\\/mpi_hello\" ]\r\nlt-flux-broker: [1442534264.833608] lwj.4.debug[1] reading lua files from /g/g12/scogland/projects/flux/flux-core/src/modules/wreck/lua.d/*.lua\r\nlt-flux-broker: [1442534264.833839] lwj.4.debug[2] lwj.4: node2: basis=2\r\nlt-flux-broker: [1442534264.834154] lwj.4.debug[3] lwj.4: node3: basis=3\r\nlt-flux-broker: [1442534264.834865] lwj.4.info[2] lwj 4: node2: nprocs=1, nnodes=4, cmdline=[ \".\\/mpi_hello\" ]\r\nlt-flux-broker: [1442534264.835122] lwj.4.debug[2] reading lua files from /g/g12/scogland/projects/flux/flux-core/src/modules/wreck/lua.d/*.lua\r\nlt-flux-broker: [1442534264.835164] lwj.4.info[3] lwj 4: node3: nprocs=1, nnodes=4, cmdline=[ \".\\/mpi_hello\" ]\r\nlt-flux-broker: [1442534264.835362] lwj.4.debug[3] reading lua files from /g/g12/scogland/projects/flux/flux-core/src/modules/wreck/lua.d/*.lua\r\nlt-flux-broker: [1442534264.837158] lwj.4.debug[1] task0: pid 113264 (./mpi_hello): started\r\nlt-flux-broker: [1442534264.838367] lwj.4.debug[2] task0: pid 77211 (./mpi_hello): started\r\nlt-flux-broker: [1442534264.838536] lwj.4.debug[3] task0: pid 121727 (./mpi_hello): started\r\nlt-flux-broker: [1442534264.840258] lwj.4.debug[0] updating job state to running\r\nlt-flux-broker: [1442534264.953990] lwj.4.debug[0] task0: pid 242448 (./mpi_hello) exited with status 0x0000\r\n0: completed MPI_Init in 0.007s.  There are 1 tasks\r\nPMGR_COLLECTIVE ERROR: rank 1 on cab22:\r\nlt-flux-broker: [1442534264.971842] lwj.4.debug[1] task0: pid 113264 (./mpi_hello) exited with status 0x008b\r\nPMGR_COLLECTIVE ERROR: rank 3 on cab32:\r\nlt-flux-broker: [1442534264.979419] lwj.4.debug[3] task0: pid 121727 (./mpi_hello) exited with status 0x008b\r\nPMGR_COLLECTIVE ERROR: rank 2 on cab31:\r\nlt-flux-broker: [1442534264.981217] lwj.4.debug[2] task0: pid 77211 (./mpi_hello) exited with status 0x008b\r\nlt-flux-broker: [1442534264.982641] lwj.4.debug[0] updating job state to complete\r\nlt-flux-broker: [1442534264.982923] lwj.4.info[1] job complete. exiting...\r\nlt-flux-broker: [1442534264.982984] lwj.4.info[2] job complete. exiting...\r\nlt-flux-broker: [1442534264.982681] lwj.4.info[3] job complete. exiting...\r\nlt-flux-broker: [1442534264.984627] lwj.4.info[0] job complete. exiting...\r\nwreckrun: tasks [1-3]: exited with signal 11\r\n0: completed first barrier\r\n0: completed MPI_Finalize\r\nlt-flux-broker: [1442534264.991765] job.info[0] got request job.disconnect\r\n```\r\n\r\nOdd that one should work and the others die, I would have thought that even if the one got the right PMI it would have been given the wrong number of ranks for COMM_WORLD and failed, but it didn't.  Explicitly setting the pmi library for preload on all nodes results in a different error:\r\n\r\n```\r\nflux wreckrun -N 4 env LD_PRELOAD=../flux-core/src/lib/libpmi/.libs/libpmi.so ./mpi_hello\r\nlt-flux-broker: [1442534350.140893] job.info[0] got request job.create\r\nlt-flux-broker: [1442534350.142364] job.info[0] Setting job 5 to reserved\r\nlt-flux-broker: waitpid: No child processes\r\nlt-flux-broker: waitpid: No child processes\r\nlt-flux-broker: [1442534350.182075] lwj.5.debug[0] initializing from CMB: rank=0\r\nlt-flux-broker: [1442534350.183149] lwj.5.debug[0] lwj.5: node0: basis=0\r\nlt-flux-broker: [1442534350.184621] lwj.5.info[0] lwj 5: node0: nprocs=1, nnodes=4, cmdline=[ \"env\", \"LD_PRELOAD=..\\/flux-core\\/src\\/lib\\/libpmi\\/.libs\\/libpmi.so\", \".\\/mpi_hello\" ]\r\nlt-flux-broker: [1442534350.184685] lwj.5.debug[0] updating job state to starting\r\nlt-flux-broker: [1442534350.185996] lwj.5.debug[0] reading lua files from /g/g12/scogland/projects/flux/flux-core/src/modules/wreck/lua.d/*.lua\r\nlt-flux-broker: [1442534350.190197] lwj.5.debug[0] task0: pid 242481 (env): started\r\nlt-flux-broker: [1442534350.229234] lwj.5.debug[1] initializing from CMB: rank=1\r\nlt-flux-broker: [1442534350.230633] lwj.5.debug[1] lwj.5: node1: basis=1\r\nlt-flux-broker: [1442534350.231312] lwj.5.info[1] lwj 5: node1: nprocs=1, nnodes=4, cmdline=[ \"env\", \"LD_PRELOAD=..\\/flux-core\\/src\\/lib\\/libpmi\\/.libs\\/libpmi.so\", \".\\/mpi_hello\" ]\r\nlt-flux-broker: [1442534350.231510] lwj.5.debug[1] reading lua files from /g/g12/scogland/projects/flux/flux-core/src/modules/wreck/lua.d/*.lua\r\nlt-flux-broker: [1442534350.241098] lwj.5.debug[1] task0: pid 113284 (env): started\r\nlt-flux-broker: [1442534350.281341] lwj.5.debug[3] initializing from CMB: rank=3\r\nlt-flux-broker: [1442534350.283600] lwj.5.debug[3] lwj.5: node3: basis=3\r\nlt-flux-broker: [1442534350.284630] lwj.5.info[3] lwj 5: node3: nprocs=1, nnodes=4, cmdline=[ \"env\", \"LD_PRELOAD=..\\/flux-core\\/src\\/lib\\/libpmi\\/.libs\\/libpmi.so\", \".\\/mpi_hello\" ]\r\nlt-flux-broker: [1442534350.284854] lwj.5.debug[3] reading lua files from /g/g12/scogland/projects/flux/flux-core/src/modules/wreck/lua.d/*.lua\r\nlt-flux-broker: [1442534350.292973] lwj.5.debug[2] initializing from CMB: rank=2\r\nlt-flux-broker: [1442534350.295110] lwj.5.debug[2] lwj.5: node2: basis=2\r\nlt-flux-broker: [1442534350.294910] lwj.5.debug[3] task0: pid 121746 (env): started\r\nlt-flux-broker: [1442534350.296157] lwj.5.info[2] lwj 5: node2: nprocs=1, nnodes=4, cmdline=[ \"env\", \"LD_PRELOAD=..\\/flux-core\\/src\\/lib\\/libpmi\\/.libs\\/libpmi.so\", \".\\/mpi_hello\" ]\r\nlt-flux-broker: [1442534350.296435] lwj.5.debug[2] reading lua files from /g/g12/scogland/projects/flux/flux-core/src/modules/wreck/lua.d/*.lua\r\nlt-flux-broker: [1442534350.306879] lwj.5.debug[2] task0: pid 77230 (env): started\r\nlt-flux-broker: [1442534350.308262] lwj.5.debug[0] updating job state to running\r\nlt-flux-broker: [1442534350.319638] lwj.5.debug[0] task0: pid 242481 (env) exited with status 0x0000\r\n0: completed MPI_Init in 0.007s.  There are 1 tasks\r\nmpi_hello: psm_entry.c:133: psm_doinit: Assertion `pg_rank < pg_size' failed.\r\nlt-flux-broker: [1442534350.368861] lwj.5.debug[1] task0: pid 113284 (env) exited with status 0x0086\r\nmpi_hello: psm_entry.c:133: psm_doinit: Assertion `pg_rank < pg_size' failed.\r\nlt-flux-broker: [1442534350.425514] lwj.5.debug[3] task0: pid 121746 (env) exited with status 0x0086\r\nmpi_hello: psm_entry.c:133: psm_doinit: Assertion `pg_rank < pg_size' failed.\r\nlt-flux-broker: [1442534350.435959] lwj.5.debug[2] task0: pid 77230 (env) exited with status 0x0086\r\nlt-flux-broker: [1442534350.437805] lwj.5.debug[0] updating job state to complete\r\nlt-flux-broker: [1442534350.437984] lwj.5.info[2] job complete. exiting...\r\nlt-flux-broker: [1442534350.438093] lwj.5.info[1] job complete. exiting...\r\nlt-flux-broker: [1442534350.437657] lwj.5.info[3] job complete. exiting...\r\nlt-flux-broker: [1442534350.439130] lwj.5.info[0] job complete. exiting...\r\nwreckrun: tasks [1-3]: exited with signal 6\r\n0: completed first barrier\r\n0: completed MPI_Finalize\r\nlt-flux-broker: [1442534350.447708] job.info[0] got request job.disconnect\r\n```\r\n\r\nThis mpi_hello is built with the dotkit mvapich2_gnu.\r\n\r\nUsing the mvapich2-gnu-psm from opt however, hereafter mpi_hello_opt, is a bit different in that both versions of the multi-node test fail the same way the second does above.  So it gets the right PMI on all nodes, then dies anyway.  It did however print a bit more of a real error message:\r\n\r\n```\r\nmpi_hello_opt: psm_entry.c:133: psm_doinit: Assertion `pg_rank < pg_size' failed.\r\nlt-flux-broker: [1442534840.676878] lwj.2.debug[2] task0: pid 77907 (env) exited with status 0x0086\r\nFatal error in MPI_Init: Internal MPI error!, error stack:\r\nlt-flux-broker: [1442534840.696838] lwj.2.debug[1] task0: pid 113961 (env) exited with status 0x0100\r\nMPIR_Init_thread(404):\r\nMPID_Init(298).......: channel initialization failed\r\n(unknown)(): Internal MPI error!\r\n```"},"comment":{"url":"https://api.github.com/repos/flux-framework/flux-core/issues/comments/141555314","html_url":"https://github.com/flux-framework/flux-core/issues/411#issuecomment-141555314","issue_url":"https://api.github.com/repos/flux-framework/flux-core/issues/411","id":141555314,"user":{"login":"grondo","id":741970,"avatar_url":"https://avatars.githubusercontent.com/u/741970?v=3","gravatar_id":"","url":"https://api.github.com/users/grondo","html_url":"https://github.com/grondo","followers_url":"https://api.github.com/users/grondo/followers","following_url":"https://api.github.com/users/grondo/following{/other_user}","gists_url":"https://api.github.com/users/grondo/gists{/gist_id}","starred_url":"https://api.github.com/users/grondo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/grondo/subscriptions","organizations_url":"https://api.github.com/users/grondo/orgs","repos_url":"https://api.github.com/users/grondo/repos","events_url":"https://api.github.com/users/grondo/events{/privacy}","received_events_url":"https://api.github.com/users/grondo/received_events","type":"User","site_admin":false},"created_at":"2015-09-18T20:15:41Z","updated_at":"2015-09-18T20:15:41Z","body":"Thanks for new title! Want to cut and paste the perf and tau outputs above into a new issue? Seems like relevant data to keep around, but unrelated to the current issues.\r\n\r\nAlso, I'm wondering if \"PMGR occasionally pretends to work\" should also be its own issue?"}}