{"action":"created","issue":{"url":"https://api.github.com/repos/irods/irods/issues/2803","labels_url":"https://api.github.com/repos/irods/irods/issues/2803/labels{/name}","comments_url":"https://api.github.com/repos/irods/irods/issues/2803/comments","events_url":"https://api.github.com/repos/irods/irods/issues/2803/events","html_url":"https://github.com/irods/irods/issues/2803","id":95503783,"number":2803,"title":"Error reported from BIND error during parallel transfer should be more verbose","user":{"login":"trel","id":55238,"avatar_url":"https://avatars.githubusercontent.com/u/55238?v=3","gravatar_id":"","url":"https://api.github.com/users/trel","html_url":"https://github.com/trel","followers_url":"https://api.github.com/users/trel/followers","following_url":"https://api.github.com/users/trel/following{/other_user}","gists_url":"https://api.github.com/users/trel/gists{/gist_id}","starred_url":"https://api.github.com/users/trel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/trel/subscriptions","organizations_url":"https://api.github.com/users/trel/orgs","repos_url":"https://api.github.com/users/trel/repos","events_url":"https://api.github.com/users/trel/events{/privacy}","received_events_url":"https://api.github.com/users/trel/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/irods/irods/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"jasoncoposky","id":1790140,"avatar_url":"https://avatars.githubusercontent.com/u/1790140?v=3","gravatar_id":"","url":"https://api.github.com/users/jasoncoposky","html_url":"https://github.com/jasoncoposky","followers_url":"https://api.github.com/users/jasoncoposky/followers","following_url":"https://api.github.com/users/jasoncoposky/following{/other_user}","gists_url":"https://api.github.com/users/jasoncoposky/gists{/gist_id}","starred_url":"https://api.github.com/users/jasoncoposky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jasoncoposky/subscriptions","organizations_url":"https://api.github.com/users/jasoncoposky/orgs","repos_url":"https://api.github.com/users/jasoncoposky/repos","events_url":"https://api.github.com/users/jasoncoposky/events{/privacy}","received_events_url":"https://api.github.com/users/jasoncoposky/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/irods/irods/milestones/12","html_url":"https://github.com/irods/irods/milestones/4.2.0","labels_url":"https://api.github.com/repos/irods/irods/milestones/12/labels","id":925749,"number":12,"title":"4.2.0","description":"","creator":{"login":"trel","id":55238,"avatar_url":"https://avatars.githubusercontent.com/u/55238?v=3","gravatar_id":"","url":"https://api.github.com/users/trel","html_url":"https://github.com/trel","followers_url":"https://api.github.com/users/trel/followers","following_url":"https://api.github.com/users/trel/following{/other_user}","gists_url":"https://api.github.com/users/trel/gists{/gist_id}","starred_url":"https://api.github.com/users/trel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/trel/subscriptions","organizations_url":"https://api.github.com/users/trel/orgs","repos_url":"https://api.github.com/users/trel/repos","events_url":"https://api.github.com/users/trel/events{/privacy}","received_events_url":"https://api.github.com/users/trel/received_events","type":"User","site_admin":false},"open_issues":105,"closed_issues":28,"state":"open","created_at":"2015-01-07T16:14:58Z","updated_at":"2015-08-25T16:11:37Z","due_on":null,"closed_at":null},"comments":6,"created_at":"2015-07-16T19:07:08Z","updated_at":"2015-08-26T11:32:57Z","closed_at":null,"body":"From https://groups.google.com/forum/#!topic/irod-chat/ylqMd3pYsTw:\r\n\r\n> we think the error message is due to the checksum being the last thing that failed, but triggered by the lower level problem of not having enough ports open (a BIND error when trying to find an open port for the large file transfer).  We'll work on making the error messages better so clients and servers can figure out what is happening.\r\n\r\nA possible additional solution may include a retry with exponential backoff."},"comment":{"url":"https://api.github.com/repos/irods/irods/issues/comments/134957161","html_url":"https://github.com/irods/irods/issues/2803#issuecomment-134957161","issue_url":"https://api.github.com/repos/irods/irods/issues/2803","id":134957161,"user":{"login":"keithj","id":102847,"avatar_url":"https://avatars.githubusercontent.com/u/102847?v=3","gravatar_id":"","url":"https://api.github.com/users/keithj","html_url":"https://github.com/keithj","followers_url":"https://api.github.com/users/keithj/followers","following_url":"https://api.github.com/users/keithj/following{/other_user}","gists_url":"https://api.github.com/users/keithj/gists{/gist_id}","starred_url":"https://api.github.com/users/keithj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/keithj/subscriptions","organizations_url":"https://api.github.com/users/keithj/orgs","repos_url":"https://api.github.com/users/keithj/repos","events_url":"https://api.github.com/users/keithj/events{/privacy}","received_events_url":"https://api.github.com/users/keithj/received_events","type":"User","site_admin":false},"created_at":"2015-08-26T11:32:57Z","updated_at":"2015-08-26T11:32:57Z","body":"The corrected script is below. (I had given `parallel` a `--halt now,fail=1` argument so that it killed all remaining jobs after the first failure.)\r\n\r\nI'm still seeing a lot of failures at 64 clients and above, but I can now see that they are due to client connection failures under load. That is not the type of failure I'm interested in and is confounding measurements.\r\n\r\nScripts adapted from the Google groups post:\r\n\r\n````\r\n#!/bin/bash\r\n\r\nmkdir random_files.dir\r\n\r\ncd random_files.dir\r\n\r\nfor i in `seq -w 256`\r\ndo\r\n    dd if=/dev/urandom of=random_file.128M.$i bs=1M count=128\r\ndone\r\n\r\nmd5sum random_file.128M.* > ../random_files.md5\r\n````\r\n\r\n````\r\n#!/bin/bash\r\n\r\nRESOURCE=demoResc\r\nN=256 # num files\r\n\r\nmkdir -p benchmarks.dir\r\nimkdir -p benchmarks.coll\r\n\r\ncd random_files.dir\r\n\r\nfor i in random_file.128M.*\r\ndo\r\n    echo iput -R ${RESOURCE} -K -f $PWD/$i benchmarks.coll/\r\ndone > ../benchmarks.dir/iput-all\r\n\r\nfor i in random_file.128M.*\r\ndo\r\n    echo iget -R ${RESOURCE} -f benchmarks.coll/$i\r\ndone > ../benchmarks.dir/iget-all\r\n\r\n# Benchmark\r\ncd ../benchmarks.dir\r\n\r\nfor j in `seq 1 3`\r\ndo\r\n    for i in `seq 32 16 $N`\r\n    do\r\n        echo\r\n        echo $i iput jobs:\r\n        date\r\n        cat iput-all | ../parallel --jobs $i --joblog iput.$i.$j.log\r\n        date\r\n        echo $i iget jobs:\r\n        date\r\n        cat iget-all | ../parallel --jobs $i --joblog iget.$i.$j.log\r\n        date\r\n        md5sum random_file.128M.* > /tmp/benchmark.md5\r\n        rm -f random_file.128M.*\r\n        echo Checksum errors: `comm -2 -3 <(sort ../random_files.md5) <(sort /tmp/benchmark.md5) | wc -l`\r\n        sleep 5\r\n    done 2>&1 | tee benchmark.$j.log\r\ndone\r\n````\r\n"}}