{"push_id":770793536,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"dc814f30daf0c80571389e7a9b6938c31ae6c1e2","before":"612957f0579faf6f8b2d2d846670f39395661e74","commits":[{"sha":"dc814f30daf0c80571389e7a9b6938c31ae6c1e2","author":{"email":"bdc985dd5c69fa6eaf7fddbd8038139ba1a2f1fe@ph.ed.ac.uk","name":"Peter Boyle"},"message":"Binary IO file for generic Grid array parallel I/O.\n\nNumber of IO MPI tasks can be varied by selecting which\ndimensions use parallel IO and which dimensions use Serial send to boss\nI/O.\n\nThus can neck down from, say 1024 nodes = 4x4x8x8 to {1,8,32,64,128,256,1024} nodes\ndoing the I/O.\n\nInterpolates nicely between ALL nodes write their data, a single boss per time-plane\nin processor space [old UKQCD fortran code did this], and a single node doing all I/O.\n\nNot sure I have the transfer sizes big enough and am not overly convinced fstream\nis guaranteed to not give buffer inconsistencies unless I set streambuf size to zero.\n\nPractically it has worked on 8 tasks, 2x1x2x2 writing /cloning NERSC configurations\non my MacOS + OpenMPI and Clang environment.\n\nIt is VERY easy to switch to pwrite at a later date, and also easy to send x-strips around from\neach node in order to gather bigger chunks at the syscall level.\n\nThat would push us up to the circa 8x 18*4*8 == 4KB size write chunk, and by taking, say, x/y non\nparallel we get to 16MB contiguous chunks written in multi 4KB transactions\nper IOnode in 64^3 lattices for configuration I/O.\n\nI suspect this is fine for system performance.","distinct":true,"url":"https://api.github.com/repos/paboyle/Grid/commits/dc814f30daf0c80571389e7a9b6938c31ae6c1e2"}]}