{"action":"created","issue":{"url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/146","labels_url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/146/labels{/name}","comments_url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/146/comments","events_url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/146/events","html_url":"https://github.com/radical-cybertools/ExTASY/issues/146","id":57797129,"number":146,"title":"Failure of Amber/CoCo on Stampede when numIterations = 9","user":{"login":"ashkurti","id":6951871,"avatar_url":"https://avatars.githubusercontent.com/u/6951871?v=3","gravatar_id":"","url":"https://api.github.com/users/ashkurti","html_url":"https://github.com/ashkurti","followers_url":"https://api.github.com/users/ashkurti/followers","following_url":"https://api.github.com/users/ashkurti/following{/other_user}","gists_url":"https://api.github.com/users/ashkurti/gists{/gist_id}","starred_url":"https://api.github.com/users/ashkurti/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ashkurti/subscriptions","organizations_url":"https://api.github.com/users/ashkurti/orgs","repos_url":"https://api.github.com/users/ashkurti/repos","events_url":"https://api.github.com/users/ashkurti/events{/privacy}","received_events_url":"https://api.github.com/users/ashkurti/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-02-16T12:13:45Z","updated_at":"2015-02-16T16:39:08Z","closed_at":null,"body":"extasy log at: https://gist.github.com/ashkurti/69a32188159e0f4d81ae\r\n\r\nAccording to the current specific requirement if numIterations = 9 we would expect 4 folders (iter2, iter4, iter6, iter8) in the backup folder containing the corresponding trajectory and coco log files. \r\n\r\nIn this case I notice only the iter2, iter4 and iter6 folders but not the iter8 folder. The run took not more than 20 minutes, therefore the walltime should not be a problem this time."},"comment":{"url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/comments/74536338","html_url":"https://github.com/radical-cybertools/ExTASY/issues/146#issuecomment-74536338","issue_url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/146","id":74536338,"user":{"login":"ashkurti","id":6951871,"avatar_url":"https://avatars.githubusercontent.com/u/6951871?v=3","gravatar_id":"","url":"https://api.github.com/users/ashkurti","html_url":"https://github.com/ashkurti","followers_url":"https://api.github.com/users/ashkurti/followers","following_url":"https://api.github.com/users/ashkurti/following{/other_user}","gists_url":"https://api.github.com/users/ashkurti/gists{/gist_id}","starred_url":"https://api.github.com/users/ashkurti/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ashkurti/subscriptions","organizations_url":"https://api.github.com/users/ashkurti/orgs","repos_url":"https://api.github.com/users/ashkurti/repos","events_url":"https://api.github.com/users/ashkurti/events{/privacy}","received_events_url":"https://api.github.com/users/ashkurti/received_events","type":"User","site_admin":false},"created_at":"2015-02-16T16:39:08Z","updated_at":"2015-02-16T16:39:08Z","body":"Investigating files created on stampede, at /work/02998/ardi/radical.pilot.sandbox/pilot-54e1d9f8f8cdba3f96beee22/unit-54e1dcacf8cdba3f96beef01:\r\n\r\n```\r\n$ ls\r\nlogfile  mdshort.in  min6.out   radical_pilot_cu_launch_script-h3QEDO.sh  STDOUT\r\nmd6.crd  min6.crd    min.in     radical_pilot_cu_launch_script-KCunzS.sh\r\nmd6.out  min6.inf    penta.top  STDERR\r\n```\r\n```\r\n$ more STDERR\r\nln: creating hard link `./STDERR': File exists\r\nln: creating hard link `./STDOUT': File exists\r\n[cli_0]: aborting job:\r\napplication called MPI_Abort(MPI_COMM_WORLD, 1) - process 0\r\n[c403-002.stampede.tacc.utexas.edu:mpispawn_0][readline] Unexpected End-Of-File on file descriptor 6. MPI process died?\r\n[c403-002.stampede.tacc.utexas.edu:mpispawn_0][mtpmi_processops] Error while reading PMI socket. MPI process died?\r\n[c403-002.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 0, pid: 86946) exited with status 1\r\nln: accessing `md6.ncdf': No such file or directory\r\n[c403-002.stampede.tacc.utexas.edu:mpispawn_0][report_error] connect() failed:  (111)\r\n```\r\n\r\n```\r\n$ more STDOUT \r\nTACC: Starting up job 4865579\r\nTACC: Setting up parallel environment for MVAPICH2+mpispawn.\r\nTACC: Starting parallel tasks...\r\nTACC: MPI job exited with code: 1\r\n \r\nTACC: Shutdown complete. Exiting.\r\n```\r\n\r\n```\r\n$ more logfile \r\n\r\nMajor Routine Parallel Profiling - NonSetup CPU Seconds:\r\n\r\n           D                                                              \r\n           a                                                              \r\n           t                                                              \r\n           a                              D                               \r\n           D         N                    i                               \r\n           i         o                    h                               \r\n           s         n             A      e      S      R      O         T\r\n   T       t         B      B      n      d      h      u      t         o\r\n   a       r         o      o      g      r      a      n      h         t\r\n   s       i         n      n      l      a      k      M      e         a\r\n   k       b         d      d      e      l      e      D      r         l\r\n------------------------------------------------------------------------------\r\n   0     0.0       0.0    0.0    0.0    0.0    0.0    0.0    0.0       0.0\r\n   1     0.0       0.0    0.0    0.0    0.0    0.0    0.0    0.0       0.0\r\n------------------------------------------------------------------------------\r\n avg     0.0       0.0    0.0    0.0    0.0    0.0    0.0    0.0       0.0\r\n min     0.0       0.0    0.0    0.0    0.0    0.0    0.0    0.0       0.0\r\n max     0.0       0.0    0.0    0.0    0.0    0.0    0.0    0.0       0.0\r\n std     0.0       0.0    0.0    0.0    0.0    0.0    0.0    0.0       0.0\r\n------------------------------------------------------------------------------\r\n\r\nGB NonBond Parallel Profiling - NonSetup CPU Seconds:\r\n\r\n                                 O         D          \r\n                                 f         i          \r\n             R                   f         s         T\r\n   T         a         D         D         t         o\r\n   a         d         i         i         r         t\r\n   s         i         a         a         i         a\r\n   k         i         g         g         b         l\r\n------------------------------------------------------\r\n   0       0.0       0.0       0.0       0.0       0.0\r\n   1       0.0       0.0       0.0       0.0       0.0\r\n------------------------------------------------------\r\n avg       0.0       0.0       0.0       0.0       0.0\r\n\r\n```\r\n\r\n```\r\n$ more radical_pilot_cu_launch_script-h3QEDO.sh\r\n#!/bin/bash -l\r\ncd /work/02998/ardi/radical.pilot.sandbox/pilot-54e1d9f8f8cdba3f96beee22/unit-54e1dcacf8cdba3f96beef01\r\nln /work/02998/ardi/radical.pilot.sandbox/pilot-54e1d9f8f8cdba3f96beee22/unit-54e1dc9ff8cdba3f96beeef1/* .\r\nmodule load TACC && module load amber\r\nln /work/02998/ardi/radical.pilot.sandbox/pilot-54e1d9f8f8cdba3f96beee22/unit-54e1da00f8cdba3f96beee24/mdshort.in .\r\n\r\n\r\n/usr/local/bin/ibrun -n 2 -o 8 pmemd.MPI \"-O\" \"-i\" \"mdshort.in\" \"-o\" \"md6.out\" \"-inf\" \"md6.inf\" \"-x\" \"md6.ncdf\" \"-r\" \"md6.rst\" \"-p\"\r\n \"penta.top\" \"-c\" \"md6.crd\"\r\nln md6.ncdf /work/02998/ardi/radical.pilot.sandbox/pilot-54e1d9f8f8cdba3f96beee22/unit-54e1dc48f8cdba3f96beeeef/md_6_0.ncdf\r\n```\r\n\r\n```\r\n$ more radical_pilot_cu_launch_script-KCunzS.sh\r\n#!/bin/bash -l\r\ncd /work/02998/ardi/radical.pilot.sandbox/pilot-54e1d9f8f8cdba3f96beee22/unit-54e1dc9ff8cdba3f96beeef1\r\nmodule load TACC && module load amber\r\nln /work/02998/ardi/radical.pilot.sandbox/pilot-54e1d9f8f8cdba3f96beee22/unit-54e1dc48f8cdba3f96beeeef/min60.crd min6.crd\r\nln /work/02998/ardi/radical.pilot.sandbox/pilot-54e1d9f8f8cdba3f96beee22/unit-54e1da00f8cdba3f96beee24/penta.top .\r\nln /work/02998/ardi/radical.pilot.sandbox/pilot-54e1d9f8f8cdba3f96beee22/unit-54e1da00f8cdba3f96beee24/min.in .\r\n\r\n\r\n/usr/local/bin/ibrun -n 2 -o 6 pmemd.MPI \"-O\" \"-i\" \"min.in\" \"-o\" \"min6.out\" \"-inf\" \"min6.inf\" \"-r\" \"md6.crd\" \"-p\" \"penta.top\" \"-c\" \r\n\"min6.crd\" \"-ref\" \"min6.crd\"\r\n```\r\n\r\n```\r\n$ ls -l /work/02998/ardi/radical.pilot.sandbox/pilot-54e1d9f8f8cdba3f96beee22/unit-54e1dc48f8cdba3f96beeeef/\r\ntotal 1800\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_0.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_10.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_11.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_12.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_13.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_14.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_15.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_1.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_2.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_3.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_4.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_5.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:03 md_5_6.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_7.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_8.ncdf\r\n-rw------- 1 ardi G-801782 71680 Feb 16 06:02 md_5_9.ncdf\r\n-rw------- 2 ardi G-801782 71680 Feb 16 06:04 md_6_13.ncdf\r\n-rw------- 2 ardi G-801782 71680 Feb 16 06:04 md_6_14.ncdf\r\n-rw------- 2 ardi G-801782 71680 Feb 16 06:04 md_6_15.ncdf\r\n-rw------- 2 ardi G-801782 71680 Feb 16 06:04 md_6_1.ncdf\r\n-rw------- 2 ardi G-801782 71680 Feb 16 06:04 md_6_2.ncdf\r\n-rw------- 2 ardi G-801782 71680 Feb 16 06:04 md_6_5.ncdf\r\n-rw------- 2 ardi G-801782 71680 Feb 16 06:04 md_6_6.ncdf\r\n-rw------- 2 ardi G-801782 71680 Feb 16 06:04 md_6_7.ncdf\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min60.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min610.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min611.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min612.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min613.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min614.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min615.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min61.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min62.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min63.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min64.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min65.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min66.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min67.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min68.crd\r\n-rw------- 4 ardi G-801782  2174 Feb 16 06:03 min69.crd\r\n-rwx------ 1 ardi G-801782   135 Feb 16 06:03 radical_pilot_cu_launch_script-_MaWUV.sh\r\n-rw------- 1 ardi G-801782     0 Feb 16 06:03 STDERR\r\n-rw------- 1 ardi G-801782     2 Feb 16 06:03 STDOUT\r\n```\r\n\r\n```\r\n$ more  /work/02998/ardi/radical.pilot.sandbox/pilot-54e1d9f8f8cdba3f96beee22/unit-54e1dc48f8cdba3f96beeeef/STDERR\r\n$ more /work/02998/ardi/radical.pilot.sandbox/pilot-54e1d9f8f8cdba3f96beee22/unit-54e1dc48f8cdba3f96beeeef/STDOUT\r\n2\r\n$ more /work/02998/ardi/radical.pilot.sandbox/pilot-54e1d9f8f8cdba3f96beee22/unit-54e1dc48f8cdba3f96beeeef/radical_pilot_cu_launch_script-_MaWUV.sh\r\n#!/bin/bash -l\r\ncd /work/02998/ardi/radical.pilot.sandbox/pilot-54e1d9f8f8cdba3f96beee22/unit-54e1dc48f8cdba3f96beeeef\r\n\r\n/bin/echo \"2\"\r\n\r\n```"}}