{"action":"created","issue":{"url":"https://api.github.com/repos/ofiwg/libfabric/issues/1195","labels_url":"https://api.github.com/repos/ofiwg/libfabric/issues/1195/labels{/name}","comments_url":"https://api.github.com/repos/ofiwg/libfabric/issues/1195/comments","events_url":"https://api.github.com/repos/ofiwg/libfabric/issues/1195/events","html_url":"https://github.com/ofiwg/libfabric/issues/1195","id":99612405,"number":1195,"title":"OFI+Sockets causes oversubscription and application crash.","user":{"login":"alexander-sannikov","id":1032158,"avatar_url":"https://avatars.githubusercontent.com/u/1032158?v=3","gravatar_id":"","url":"https://api.github.com/users/alexander-sannikov","html_url":"https://github.com/alexander-sannikov","followers_url":"https://api.github.com/users/alexander-sannikov/followers","following_url":"https://api.github.com/users/alexander-sannikov/following{/other_user}","gists_url":"https://api.github.com/users/alexander-sannikov/gists{/gist_id}","starred_url":"https://api.github.com/users/alexander-sannikov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexander-sannikov/subscriptions","organizations_url":"https://api.github.com/users/alexander-sannikov/orgs","repos_url":"https://api.github.com/users/alexander-sannikov/repos","events_url":"https://api.github.com/users/alexander-sannikov/events{/privacy}","received_events_url":"https://api.github.com/users/alexander-sannikov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-08-07T09:19:04Z","updated_at":"2015-08-10T20:27:28Z","closed_at":null,"body":"Usage of OFI with socket provider causes to application crash.\r\nWhen I launch workload withOFI:Sockets on all available cores it is leads to crash on different MPI versions. \r\nScale threshold leading to fault is differs from  one MPI to another. In some cases it is reproducing on smaller scales. Reproducer and traces are provided below. They are will be reproduced  with MPICH 3.2b3 and both libfabric versions (v1.0.0 and 1.1.0).  \r\n\r\n```bash\r\n ~> mpirun -n $((`cpuinfo|grep Cores|head -n 1|awk '{print $3}'` * `cat ${HYDRA_HOST_FILE}|wc -l`/2)) ./a.out\r\nSuccess!\r\n ~> mpirun -n $((`cpuinfo|grep Cores|head -n 1|awk '{print $3}'` * `cat ${HYDRA_HOST_FILE}|wc -l`)) ./a.out\r\nFatal error in PMPI_Isend: Other MPI error, error stack:\r\nPMPI_Isend(149): MPI_Isend(buf=0x7fffb8e81e90, count=4096, MPI_BYTE, dest=0, tag=10, MPI_COMM_WORLD, request=0x7fffb8e81e8c) failed\r\ndo_isend(193)..: OFI tagged sendto failed (ofi_tagged_template.c:193:do_isend:Connection reset by peer)\r\nFatal error in PMPI_Isend: Other MPI error, error stack:\r\nPMPI_Isend(149): MPI_Isend(buf=0x7fff1c693090, count=4096, MPI_BYTE, dest=0, tag=10, MPI_COMM_WORLD, request=0x7fff1c69308c) failed\r\ndo_isend(193)..: OFI tagged sendto failed (ofi_tagged_template.c:193:do_isend:Connection reset by peer)\r\nFatal error in PMPI_Isend: Other MPI error, error stack:\r\nPMPI_Isend(149): MPI_Isend(buf=0x7fffb0aa9090, count=4096, MPI_BYTE, dest=0, tag=10, MPI_COMM_WORLD, request=0x7fffb0aa908c) failed\r\ndo_isend(193)..: OFI tagged sendto failed (ofi_tagged_template.c:193:do_isend:Connection reset by peer)\r\nFatal error in PMPI_Isend: Other MPI error, error stack:\r\nPMPI_Isend(149): MPI_Isend(buf=0x7fffcbea0c90, count=4096, MPI_BYTE, dest=0, tag=10, MPI_COMM_WORLD, request=0x7fffcbea0c8c) failed\r\ndo_isend(193)..: OFI tagged sendto failed (ofi_tagged_template.c:193:do_isend:Connection reset by peer)\r\nFatal error in PMPI_Isend: Other MPI error, error stack:\r\nPMPI_Isend(149): MPI_Isend(buf=0x7fff561e5990, count=4096, MPI_BYTE, dest=0, tag=10, MPI_COMM_WORLD, request=0x7fff561e598c) failed\r\ndo_isend(193)..: OFI tagged sendto failed (ofi_tagged_template.c:193:do_isend:Connection reset by peer)\r\n\r\n```\r\n\r\nHyda host file:\r\n> node01:1\r\n> node02:1\r\n\r\nReproducer:\r\n```C\r\n#include <mpi.h>\r\n#include <stdio.h>\r\n\r\nint main(int argc, char *argv[])\r\n{\r\n    int i = 0, rank;\r\n    int skip, numprocs;\r\n    int nsends;\r\n\r\n    MPI_Init(&argc, &argv);\r\n    MPI_Comm_rank(MPI_COMM_WORLD, &rank);\r\n    MPI_Comm_size(MPI_COMM_WORLD, &numprocs);\r\n    \r\n    int dest;\r\n    MPI_Request rr;\r\n    MPI_Status st;\r\n    \r\n    char buf[4096] = {0};\r\n    char pname[256] = {0};\r\n    int len=256;\r\n    MPI_Get_processor_name(pname,&len);\r\n    int remain = 4095;\r\n    remain -= snprintf(buf,remain,\"Rank[%d] on %s ->\",rank,pname);\r\n    \r\n    for(i=0;i<(int)(numprocs/2);i++){\r\n        dest = i*2 + !(rank%2);\r\n        remain -= snprintf(buf+4095-remain,remain,\"%d\\t\",dest);\r\n    }\r\n    fprintf(stderr, \"%s\\n\",buf);\r\n    for (i=0;i<=4096;i++) buf[i]=0;\r\n    \r\n    for(nsends = 10000; nsends>0; nsends--){\r\n        for (i=0;i<(int)(numprocs/2);i++){\r\n            dest = i*2 + !(rank%2);\r\n            MPI_Isend((void*)buf, 4096, MPI_BYTE, dest,10, MPI_COMM_WORLD, &rr);   \r\n        }\r\n        MPI_Barrier(MPI_COMM_WORLD);\r\n        for (i=0;i<(int)(numprocs/2);i++){\r\n            MPI_Recv((void*)buf, 4096, MPI_BYTE, MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &st);   \r\n        }\r\n    }\r\n    if (rank==0)fprintf(stderr,\"Success!\\n\");\r\n    MPI_Finalize();\r\n    return 0;\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/ofiwg/libfabric/issues/comments/129596477","html_url":"https://github.com/ofiwg/libfabric/issues/1195#issuecomment-129596477","issue_url":"https://api.github.com/repos/ofiwg/libfabric/issues/1195","id":129596477,"user":{"login":"jsquyres","id":617316,"avatar_url":"https://avatars.githubusercontent.com/u/617316?v=3","gravatar_id":"","url":"https://api.github.com/users/jsquyres","html_url":"https://github.com/jsquyres","followers_url":"https://api.github.com/users/jsquyres/followers","following_url":"https://api.github.com/users/jsquyres/following{/other_user}","gists_url":"https://api.github.com/users/jsquyres/gists{/gist_id}","starred_url":"https://api.github.com/users/jsquyres/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsquyres/subscriptions","organizations_url":"https://api.github.com/users/jsquyres/orgs","repos_url":"https://api.github.com/users/jsquyres/repos","events_url":"https://api.github.com/users/jsquyres/events{/privacy}","received_events_url":"https://api.github.com/users/jsquyres/received_events","type":"User","site_admin":false},"created_at":"2015-08-10T20:27:27Z","updated_at":"2015-08-10T20:27:27Z","body":"@charlesarcher :+1:  I missed that.  :-)"}}