{"action":"created","issue":{"url":"https://api.github.com/repos/cBio/cbio-cluster/issues/256","labels_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/256/labels{/name}","comments_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/256/comments","events_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/256/events","html_url":"https://github.com/cBio/cbio-cluster/issues/256","id":76378285,"number":256,"title":"MPI program failure -- inconsistent exit codes","user":{"login":"elzbth","id":6425383,"avatar_url":"https://avatars.githubusercontent.com/u/6425383?v=3","gravatar_id":"","url":"https://api.github.com/users/elzbth","html_url":"https://github.com/elzbth","followers_url":"https://api.github.com/users/elzbth/followers","following_url":"https://api.github.com/users/elzbth/following{/other_user}","gists_url":"https://api.github.com/users/elzbth/gists{/gist_id}","starred_url":"https://api.github.com/users/elzbth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elzbth/subscriptions","organizations_url":"https://api.github.com/users/elzbth/orgs","repos_url":"https://api.github.com/users/elzbth/repos","events_url":"https://api.github.com/users/elzbth/events{/privacy}","received_events_url":"https://api.github.com/users/elzbth/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":67,"created_at":"2015-05-14T14:36:39Z","updated_at":"2015-05-22T17:13:31Z","closed_at":null,"body":"Hi all, \r\n\r\n   I am trying to run an MPI program, SMUFIN, for alignment-free variant calling. It runs fine on the test dataset but on any of mine it fails. \r\nOn a large dataset, it fails after ~2days with exit code 9. \r\nOn a smaller test dataset, it fails after a minute with exit code 1. \r\n\r\nIf anybody would have thoughts about how to start figuring this out, that would be great!! \r\n\r\n** pasted below is the output in the first case (large dataset) ** \r\n\r\n\r\n...\r\nnum_targets: 25\r\n\r\nnum_targets: 25\r\n\r\nnum_targets: 25\r\n\r\nnum_targets: 25\r\n\r\nnum_targets: 25\r\n\r\nnum_targets: 25\r\n\r\nnum_targets: 25\r\n\r\nnum_targets: 25\r\n\r\n\r\n\r\n===================================================================================\r\n\r\n=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES\r\n\r\n=   EXIT CODE: 9\r\n\r\n=   CLEANING UP REMAINING PROCESSES\r\n\r\n=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES\r\n\r\n===================================================================================\r\n\r\n[proxy:0:0@cpu-6-2.local] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:883): assert (!closed) failed\r\n\r\n[proxy:0:0@cpu-6-2.local] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status\r\n\r\n[proxy:0:0@cpu-6-2.local] main (./pm/pmiserv/pmip.c:210): demux engine error waiting for event\r\n\r\n[mpiexec@cpu-6-2.local] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting\r\n\r\n[mpiexec@cpu-6-2.local] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion\r\n\r\n[mpiexec@cpu-6-2.local] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:216): launcher returned error waiting for completion\r\n\r\n[mpiexec@cpu-6-2.local] main (./ui/mpich/mpiexec.c:325): process manager error waiting for completion\r\n\r\n...\r\n\r\nThe full output file is here:\r\n\r\n/cbio/ski/kentsis/home/henaffe/smufin_NYGC.8c.128g.o3194763\r\n\r\nand the script I used to submit it is here:\r\n\r\n/cbio/ski/kentsis/home/henaffe/scripts/torque-submit-smufin-NYGC.8.128.bash\r\n\r\n**pasted below is the output for the second case (small dataset) ** \r\n\r\n...\r\nnum_targets: 25\r\n\r\nnum_targets: 25\r\n\r\nFatal error in MPI_Send: Invalid rank, error stack:\r\n\r\nMPI_Send(171): MPI_Send(buf=0x300e18fee8, count=40, MPI_CHAR, dest=12742352, tag=8, MPI_COMM_WORLD) failed\r\n\r\nMPI_Send(97).: Invalid rank has value 12742352 but must be nonnegative and less than 8\r\n\r\n\r\n\r\n===================================================================================\r\n\r\n=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES\r\n\r\n=   EXIT CODE: 1\r\n\r\n=   CLEANING UP REMAINING PROCESSES\r\n\r\n=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES\r\n\r\n===================================================================================\r\n\r\nWed May 13 14:00:08 EDT 2015\r\n\r\n\r\nthe files are:\r\nscript:\r\n\r\n/cbio/ski/kentsis/home/henaffe/scripts/torque-submit-test-rhabdoid.sh\r\n\r\noutput:\r\n\r\n\r\n/cbio/ski/kentsis/home/henaffe/smufin_NYGC.8c.4g.o3238657\r\n\r\n\r\nAny help greatly appreciated!!! \r\n\r\nCheers, \r\n\r\nElizabeth\r\n"},"comment":{"url":"https://api.github.com/repos/cBio/cbio-cluster/issues/comments/104717578","html_url":"https://github.com/cBio/cbio-cluster/issues/256#issuecomment-104717578","issue_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/256","id":104717578,"user":{"login":"elzbth","id":6425383,"avatar_url":"https://avatars.githubusercontent.com/u/6425383?v=3","gravatar_id":"","url":"https://api.github.com/users/elzbth","html_url":"https://github.com/elzbth","followers_url":"https://api.github.com/users/elzbth/followers","following_url":"https://api.github.com/users/elzbth/following{/other_user}","gists_url":"https://api.github.com/users/elzbth/gists{/gist_id}","starred_url":"https://api.github.com/users/elzbth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elzbth/subscriptions","organizations_url":"https://api.github.com/users/elzbth/orgs","repos_url":"https://api.github.com/users/elzbth/repos","events_url":"https://api.github.com/users/elzbth/events{/privacy}","received_events_url":"https://api.github.com/users/elzbth/received_events","type":"User","site_admin":false},"created_at":"2015-05-22T17:13:31Z","updated_at":"2015-05-22T17:13:31Z","body":"hmm, interesting ... I do know that in some previous jobs I had requested\nmem=896 and those had been scheduled, so maybe mem does refer to the\ncombined memory over all the nodes?\n\nI believe those jobs were\n\n3194763\nand\n3264864\n\nah yes and thanks for the correction, it is indeed not an OOM :)\n\nCheers,\n\nElizabeth\n\nOn Fri, May 22, 2015 at 1:11 PM, tatarsky <notifications@github.com> wrote:\n\n> So I've tried to explain we don't want the queue default mem=1G involved\n> at all in these MPI jobs (or really any mem= limit) because we are clearly\n> telling it our per-process memory (pmem=).\n>\n> But when we specify that value larger we don't get scheduled on the nodes.\n> So we're missing something here but I'm not clear on how to solve it from\n> the docs.\n>\n> I do see various writeups of MPI and torque that clearly show *JUST* the\n> pmem limit being used and I'm beginning to suspect their queue doesn't set\n> a mem= default....\n>\n> Case = 22238.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/cBio/cbio-cluster/issues/256#issuecomment-104716621>.\n>\n"}}