{"action":"created","issue":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/473","labels_url":"https://api.github.com/repos/open-mpi/ompi/issues/473/labels{/name}","comments_url":"https://api.github.com/repos/open-mpi/ompi/issues/473/comments","events_url":"https://api.github.com/repos/open-mpi/ompi/issues/473/events","html_url":"https://github.com/open-mpi/ompi/issues/473","id":61259995,"number":473,"title":"Vader BTL crashes","user":{"login":"jsquyres","id":617316,"avatar_url":"https://avatars.githubusercontent.com/u/617316?v=3","gravatar_id":"","url":"https://api.github.com/users/jsquyres","html_url":"https://github.com/jsquyres","followers_url":"https://api.github.com/users/jsquyres/followers","following_url":"https://api.github.com/users/jsquyres/following{/other_user}","gists_url":"https://api.github.com/users/jsquyres/gists{/gist_id}","starred_url":"https://api.github.com/users/jsquyres/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsquyres/subscriptions","organizations_url":"https://api.github.com/users/jsquyres/orgs","repos_url":"https://api.github.com/users/jsquyres/repos","events_url":"https://api.github.com/users/jsquyres/events{/privacy}","received_events_url":"https://api.github.com/users/jsquyres/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/open-mpi/ompi/labels/blocker","name":"blocker","color":"e11d21"},{"url":"https://api.github.com/repos/open-mpi/ompi/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"hjelmn","id":1226817,"avatar_url":"https://avatars.githubusercontent.com/u/1226817?v=3","gravatar_id":"","url":"https://api.github.com/users/hjelmn","html_url":"https://github.com/hjelmn","followers_url":"https://api.github.com/users/hjelmn/followers","following_url":"https://api.github.com/users/hjelmn/following{/other_user}","gists_url":"https://api.github.com/users/hjelmn/gists{/gist_id}","starred_url":"https://api.github.com/users/hjelmn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hjelmn/subscriptions","organizations_url":"https://api.github.com/users/hjelmn/orgs","repos_url":"https://api.github.com/users/hjelmn/repos","events_url":"https://api.github.com/users/hjelmn/events{/privacy}","received_events_url":"https://api.github.com/users/hjelmn/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/open-mpi/ompi/milestones/5","html_url":"https://github.com/open-mpi/ompi/milestones/Open%20MPI%201.8.5","labels_url":"https://api.github.com/repos/open-mpi/ompi/milestones/5/labels","id":833054,"number":5,"title":"Open MPI 1.8.5","description":"","creator":{"login":"jsquyres","id":617316,"avatar_url":"https://avatars.githubusercontent.com/u/617316?v=3","gravatar_id":"","url":"https://api.github.com/users/jsquyres","html_url":"https://github.com/jsquyres","followers_url":"https://api.github.com/users/jsquyres/followers","following_url":"https://api.github.com/users/jsquyres/following{/other_user}","gists_url":"https://api.github.com/users/jsquyres/gists{/gist_id}","starred_url":"https://api.github.com/users/jsquyres/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jsquyres/subscriptions","organizations_url":"https://api.github.com/users/jsquyres/orgs","repos_url":"https://api.github.com/users/jsquyres/repos","events_url":"https://api.github.com/users/jsquyres/events{/privacy}","received_events_url":"https://api.github.com/users/jsquyres/received_events","type":"User","site_admin":false},"open_issues":28,"closed_issues":20,"state":"open","created_at":"2014-10-20T17:17:04Z","updated_at":"2015-03-13T21:43:03Z","due_on":null,"closed_at":null},"comments":13,"created_at":"2015-03-13T21:43:03Z","updated_at":"2015-03-16T16:09:55Z","closed_at":null,"body":"Per the thread starting here: http://www.open-mpi.org/community/lists/devel/2015/03/17131.php\r\n\r\n@opoplawski is seeing crashes in the Open MPI test suite in openmpi-1.8.4-99-20150228 (Feb 28 nightly tarball) with the vader BTL.  If he disables the vader BTL, the crashes go away:\r\n\r\n```\r\n#0  mca_btl_vader_check_fboxes () at btl_vader_fbox.h:192\r\n#1  mca_btl_vader_component_progress () at btl_vader_component.c:694\r\n#2  0xf3971b69 in opal_progress () at runtime/opal_progress.c:187\r\n#3  0xf40b4695 in opal_condition_wait (c=<optimized out>, m=<optimized out>)\r\n   at ../opal/threads/condition.h:78\r\n#4  ompi_request_default_wait_all (count=6, requests=<optimized out>,\r\nstatuses=0x0)\r\n   at request/req_wait.c:281\r\n#5  0xf28bb5e7 in ompi_coll_tuned_alltoall_intra_basic_linear\r\n(sbuf=sbuf@entry=0xf7a2d328,\r\n   scount=scount@entry=1, sdtype=sdtype@entry=0xf4148240 <ompi_mpi_int>,\r\n   rbuf=rbuf@entry=0xf7af1920, rcount=rcount@entry=1,\r\n   rdtype=rdtype@entry=0xf4148240 <ompi_mpi_int>, comm=comm@entry=0xf7b051d8,\r\n   module=module@entry=0xf7a2b4d0) at coll_tuned_alltoall.c:700\r\n#6  0xf28b4d08 in ompi_coll_tuned_alltoall_intra_dec_fixed (sbuf=0xf7a2d328,\r\nscount=1,\r\n   sdtype=0xf4148240 <ompi_mpi_int>, rbuf=0xf7af1920, rcount=1,\r\n   rdtype=0xf4148240 <ompi_mpi_int>, comm=0xf7b051d8, module=0xf7a2b4d0)\r\n   at coll_tuned_decision_fixed.c:130\r\n#7  0xf40c7899 in PMPI_Alltoall (sendbuf=sendbuf@entry=0xf7a2d328,\r\n   sendcount=sendcount@entry=1, sendtype=sendtype@entry=0xf4148240\r\n<ompi_mpi_int>,\r\n   recvbuf=recvbuf@entry=0xf7af1920, recvcount=recvcount@entry=1,\r\n   recvtype=recvtype@entry=0xf4148240 <ompi_mpi_int>, comm=0xf7b051d8) at\r\npalltoall.c:111\r\n#8  0xe9780da0 in ADIOI_Calc_others_req (fd=fd@entry=0xf7b12640,\r\ncount_my_req_procs=1,\r\n   count_my_req_per_proc=0xf7a2d328, my_req=0xf7b00750, nprocs=4, myrank=0,\r\n   count_others_req_procs_ptr=count_others_req_procs_ptr@entry=0xffbea6e8,\r\n   others_req_ptr=others_req_ptr@entry=0xffbea6cc) at\r\nadio/common/ad_aggregate.c:453\r\n#9  0xe9796a14 in ADIOI_GEN_WriteStridedColl (fd=0xf7b12640, buf=0xf7aa0148,\r\ncount=2440,\r\n   datatype=0xf4148840 <ompi_mpi_byte>, file_ptr_type=100, offset=0,\r\nstatus=0xffbea8b8,\r\n   error_code=0xffbea790) at adio/common/ad_write_coll.c:192\r\n#10 0xe97779e0 in MPIOI_File_write_all (fh=fh@entry=0xf7b12640,\r\noffset=offset@entry=0,\r\n   file_ptr_type=file_ptr_type@entry=100, buf=buf@entry=0xf7aa0148,\r\ncount=count@entry=2440,\r\n   datatype=datatype@entry=0xf4148840 <ompi_mpi_byte>,\r\n   myname=myname@entry=0xe97a9a1c <myname.9354> \"MPI_FILE_WRITE_AT_ALL\",\r\n   status=status@entry=0xffbea8b8) at mpi-io/write_all.c:116\r\n#11 0xe9778176 in mca_io_romio_dist_MPI_File_write_at_all (fh=0xf7b12640,\r\n   offset=offset@entry=0, buf=buf@entry=0xf7aa0148, count=count@entry=2440,\r\n   datatype=datatype@entry=0xf4148840 <ompi_mpi_byte>,\r\nstatus=status@entry=0xffbea8b8)\r\n   at mpi-io/write_atall.c:55\r\n#12 0xe9770bcc in mca_io_romio_file_write_at_all (fh=0xf7aa27c8, offset=0,\r\nbuf=0xf7aa0148,\r\n   count=2440, datatype=0xf4148840 <ompi_mpi_byte>, status=0xffbea8b8)\r\n   at src/io_romio_file_write.c:61\r\n#13 0xf40ff3ce in PMPI_File_write_at_all (fh=0xf7aa27c8, offset=0,\r\nbuf=buf@entry=0xf7aa0148,\r\n   count=count@entry=2440, e=0xf4148840 <ompi_mpi_byte>,\r\nstatus=status@entry=0xffbea8b8)\r\n   at pfile_write_at_all.c:75\r\n#14 0xf437a43c in H5FD_mpio_write (_file=_file@entry=0xf7b074a8,\r\n   type=type@entry=H5FD_MEM_DRAW, dxpl_id=167772177, addr=31780,\r\nsize=size@entry=2440,\r\n   buf=buf@entry=0xf7aa0148) at ../../src/H5FDmpio.c:1840\r\n#15 0xf4375cd5 in H5FD_write (file=0xf7b074a8, dxpl=0xf7a47d20,\r\ntype=H5FD_MEM_DRAW,\r\n   addr=31780, size=size@entry=2440, buf=buf@entry=0xf7aa0148) at\r\n../../src/H5FDint.c:245\r\n#16 0xf4360932 in H5F__accum_write (fio_info=fio_info@entry=0xffbea9d4,\r\n   type=type@entry=H5FD_MEM_DRAW, addr=31780, size=size@entry=2440,\r\nbuf=buf@entry=0xf7aa0148)\r\n   at ../../src/H5Faccum.c:824\r\n#17 0xf436430c in H5F_block_write (f=0xf7a31860,\r\ntype=type@entry=H5FD_MEM_DRAW, addr=31780,\r\n   size=size@entry=2440, dxpl_id=167772177, buf=0xf7aa0148) at\r\n../../src/H5Fio.c:170\r\n#18 0xf43413ee in H5D__mpio_select_write (io_info=0xffbeab60,\r\ntype_info=0xffbeab1c,\r\n   mpi_buf_count=2440, file_space=0x0, mem_space=0x0) at ../../src/H5Dmpio.c:296\r\n#19 0xf4341f33 in H5D__final_collective_io (mpi_buf_type=0xffbeaa7c,\r\nmpi_file_type=0xffbeaa78,\r\n   mpi_buf_count=<optimized out>, type_info=0xffbeab1c, io_info=0xffbeab60)\r\n   at ../../src/H5Dmpio.c:1444\r\n#20 H5D__inter_collective_io (mem_space=0xf7a38120, file_space=0xf7a55590,\r\n   type_info=0xffbeab1c, io_info=0xffbeab60) at ../../src/H5Dmpio.c:1400\r\n#21 H5D__contig_collective_write (io_info=0xffbeab60, type_info=0xffbeab1c,\r\nnelmts=610,\r\n   file_space=0xf7a55590, mem_space=0xf7a38120, fm=0xffbeace0) at\r\n../../src/H5Dmpio.c:528\r\n#22 0xf433ae8d in H5D__write (buf=0xf7aa0148, dxpl_id=167772177,\r\nfile_space=0xf7a55590,\r\n   mem_space=0xf7a38120, mem_type_id=-140159600, dataset=0xf7a3eb40) at\r\n../../src/H5Dio.c:787\r\n#23 H5D__pre_write (dset=dset@entry=0xf7a3eb40, direct_write=<optimized out>,\r\n   mem_type_id=mem_type_id@entry=50331747, mem_space=mem_space@entry=0xf7a38120,\r\n   file_space=0xf7a55590, dxpl_id=dxpl_id@entry=167772177,\r\nbuf=buf@entry=0xf7aa0148)\r\n   at ../../src/H5Dio.c:351\r\n#24 0xf433b74c in H5Dwrite (dset_id=83886085, mem_type_id=50331747,\r\n   mem_space_id=mem_space_id@entry=67108867,\r\nfile_space_id=file_space_id@entry=67108866,\r\n   dxpl_id=dxpl_id@entry=167772177, buf=buf@entry=0xf7aa0148) at\r\n../../src/H5Dio.c:270\r\n#25 0xf466b603 in nc4_put_vara (nc=0xf7a05c58, ncid=ncid@entry=65536,\r\nvarid=varid@entry=3,\r\n   startp=startp@entry=0xffbf6a08, countp=countp@entry=0xffbf6a10,\r\n   mem_nc_type=mem_nc_type@entry=5, is_long=is_long@entry=0,\r\ndata=data@entry=0xf7a07c40)\r\n   at ../../libsrc4/nc4hdf.c:788\r\n#26 0xf4673c55 in nc4_put_vara_tc (mem_type_is_long=0, op=0xf7a07c40,\r\ncountp=0xffbf6a10,\r\n   startp=0xffbf6a08, mem_type=5, varid=3, ncid=65536) at\r\n../../libsrc4/nc4var.c:1429\r\n#27 NC4_put_vara (ncid=65536, varid=3, startp=0xffbf6a08, countp=0xffbf6a10,\r\nop=0xf7a07c40,\r\n   memtype=5) at ../../libsrc4/nc4var.c:1565\r\n#28 0xf460a377 in NC_put_vara (ncid=ncid@entry=65536, varid=varid@entry=3,\r\n   start=start@entry=0xffbf6a08, edges=edges@entry=0xffbf6a10,\r\nvalue=value@entry=0xf7a07c40,\r\n   memtype=memtype@entry=5) at ../../libdispatch/dvarput.c:79\r\n#29 0xf460b541 in nc_put_vara_float (ncid=65536, varid=3, startp=0xffbf6a08,\r\n   countp=0xffbf6a10, op=0xf7a07c40) at ../../libdispatch/dvarput.c:655\r\n#30 0xf77d06ed in test_pio_2d (cache_size=67108864, facc_type=8192,\r\naccess_flag=1,\r\n   comm=0xf414d800 <ompi_mpi_comm_world>, info=0xf4154240\r\n<ompi_mpi_info_null>, mpi_size=4,\r\n   mpi_rank=0, chunk_size=0xffbf76f4) at ../../nc_test4/tst_nc4perf.c:96\r\n#31 0xf77cfdb1 in main (argc=1, argv=0xffbf7804) at\r\n../../nc_test4/tst_nc4perf.c:299\r\n```\r\n\r\n@hjelmn Can you have a look?"},"comment":{"url":"https://api.github.com/repos/open-mpi/ompi/issues/comments/81765644","html_url":"https://github.com/open-mpi/ompi/issues/473#issuecomment-81765644","issue_url":"https://api.github.com/repos/open-mpi/ompi/issues/473","id":81765644,"user":{"login":"opoplawski","id":814662,"avatar_url":"https://avatars.githubusercontent.com/u/814662?v=3","gravatar_id":"","url":"https://api.github.com/users/opoplawski","html_url":"https://github.com/opoplawski","followers_url":"https://api.github.com/users/opoplawski/followers","following_url":"https://api.github.com/users/opoplawski/following{/other_user}","gists_url":"https://api.github.com/users/opoplawski/gists{/gist_id}","starred_url":"https://api.github.com/users/opoplawski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/opoplawski/subscriptions","organizations_url":"https://api.github.com/users/opoplawski/orgs","repos_url":"https://api.github.com/users/opoplawski/repos","events_url":"https://api.github.com/users/opoplawski/events{/privacy}","received_events_url":"https://api.github.com/users/opoplawski/received_events","type":"User","site_admin":false},"created_at":"2015-03-16T16:09:55Z","updated_at":"2015-03-16T16:09:55Z","body":"@edgargabriel  See earlier comments for what test and openmpi version this is.\r\n\r\nSome more details:\r\n\r\n The HDF5 call seems to be from ./hdf5-1.8.14/src/H5FDmpio.c:1091:\r\n\r\n        if (MPI_SUCCESS != (mpi_code=MPI_File_set_size(fh, (MPI_Offset)0)))\r\n\r\n"}}