{"action":"opened","issue":{"url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/208","labels_url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/208/labels{/name}","comments_url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/208/comments","events_url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/208/events","html_url":"https://github.com/radical-cybertools/ExTASY/issues/208","id":112689513,"number":208,"title":"Gromacs/LSDMap ExTASY 0.2 fails on ARCHER","user":{"login":"ibethune","id":7913631,"avatar_url":"https://avatars.githubusercontent.com/u/7913631?v=3","gravatar_id":"","url":"https://api.github.com/users/ibethune","html_url":"https://github.com/ibethune","followers_url":"https://api.github.com/users/ibethune/followers","following_url":"https://api.github.com/users/ibethune/following{/other_user}","gists_url":"https://api.github.com/users/ibethune/gists{/gist_id}","starred_url":"https://api.github.com/users/ibethune/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ibethune/subscriptions","organizations_url":"https://api.github.com/users/ibethune/orgs","repos_url":"https://api.github.com/users/ibethune/repos","events_url":"https://api.github.com/users/ibethune/events{/privacy}","received_events_url":"https://api.github.com/users/ibethune/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-10-21T22:09:02Z","updated_at":"2015-10-21T22:09:02Z","closed_at":null,"body":"Hi, I COCO/Amber is working OK on ARCHER using the latest ensemblemd release (0.3.6), but Gromacs/LSDMap fails on ARCHER.  I tested this from both my e290 account running from my laptop and my y14 guest account running from workflow.iu.xsede.org.  The error is the same in both cases. The relevant output from the job is:\r\n\r\n```\r\n(extasy-test)[ibethune@workflow grlsd-on-archer]$ EXTASY_DEBUG=True RADICAL_ENMD_VERBOSE=REPORT python extasy_gromacs_lsdmap.py --RPconfig archer.rcfg --Kconfig gromacslsdmap.wcfg\r\n\r\n================================================================================\r\n EnsembleMD (0.3.6)                                                             \r\n================================================================================\r\n\r\nStarting Allocation                                                           ok\r\n        Verifying pattern                                                     ok\r\n        Starting pattern execution                                            ok\r\n--------------------------------------------------------------------------------\r\nExecuting simulation-analysis loop with 1 iterations on 24 allocated core(s) on 'epsrc.archer'\r\n\r\nJob waiting on queue...\r\nJob is now running !\r\nWaiting for pre_loop step to complete.                                      done\r\nIteration 1: Waiting for simulation tasks: md.gromacs to complete           done\r\nIteration 1: Waiting for analysis tasks: md.pre_lsdmap to complete          done\r\nIteration 1: Waiting for analysis tasks: md.lsdmap to complete              done\r\nIteration 1: Waiting for analysis tasks: md.post_lsdmap to complete2015-10-21 17:48:23,976: radical.pilot       : MainProcess                     : OutputFileTransferWorker-2: ERROR   : Output transfer failed: File does not exist: '/work/y14/y14/guest02/radical.pilot.sandbox/rp.session.workflow.iu.xsede.org.ibethune.016729.0021-pilot.0000/unit.000027/out.gro' -  (/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/adaptors/shell/shell_file.py +1056 (initialize)  :  raise saga.DoesNotExist(\"File does not exist: '%s' - %s\" % (self.url.path, out)))\r\nTraceback (most recent call last):\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/radical/pilot/controller/output_file_transfer_worker.py\", line 147, in run\r\n    session=self._session)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/filesystem/file.py\", line 86, in __init__\r\n    _adaptor, _adaptor_state, _ttype=_ttype)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/namespace/entry.py\", line 89, in __init__\r\n    url, flags, session, ttype=_ttype)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/base.py\", line 101, in __init__\r\n    self._init_task = self._adaptor.init_instance (adaptor_state, *args, **kwargs)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/adaptors/cpi/decorators.py\", line 57, in wrap_function\r\n    return sync_function (self, *args, **kwargs)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/adaptors/shell/shell_file.py\", line 1007, in init_instance\r\n    self.initialize ()\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/adaptors/shell/shell_file.py\", line 1056, in initialize\r\n    raise saga.DoesNotExist(\"File does not exist: '%s' - %s\" % (self.url.path, out))\r\nDoesNotExist: File does not exist: '/work/y14/y14/guest02/radical.pilot.sandbox/rp.session.workflow.iu.xsede.org.ibethune.016729.0021-pilot.0000/unit.000027/out.gro' -  (/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/adaptors/shell/shell_file.py +1056 (initialize)  :  raise saga.DoesNotExist(\"File does not exist: '%s' - %s\" % (self.url.path, out)))\r\nException in thread OutputFileTransferWorker-2:\r\nTraceback (most recent call last):\r\n  File \"/opt/radical/lib/python2.7/threading.py\", line 810, in __bootstrap_inner\r\n    self.run()\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/radical/pilot/controller/output_file_transfer_worker.py\", line 147, in run\r\n    session=self._session)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/filesystem/file.py\", line 86, in __init__\r\n    _adaptor, _adaptor_state, _ttype=_ttype)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/namespace/entry.py\", line 89, in __init__\r\n    url, flags, session, ttype=_ttype)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/base.py\", line 101, in __init__\r\n    self._init_task = self._adaptor.init_instance (adaptor_state, *args, **kwargs)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/adaptors/cpi/decorators.py\", line 57, in wrap_function\r\n    return sync_function (self, *args, **kwargs)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/adaptors/shell/shell_file.py\", line 1007, in init_instance\r\n    self.initialize ()\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/adaptors/shell/shell_file.py\", line 1056, in initialize\r\n    raise saga.DoesNotExist(\"File does not exist: '%s' - %s\" % (self.url.path, out))\r\nDoesNotExist: File does not exist: '/work/y14/y14/guest02/radical.pilot.sandbox/rp.session.workflow.iu.xsede.org.ibethune.016729.0021-pilot.0000/unit.000027/out.gro' -  (/home/ibethune/extasy-test/lib/python2.7/site-packages/saga/adaptors/shell/shell_file.py +1056 (initialize)  :  raise saga.DoesNotExist(\"File does not exist: '%s' - %s\" % (self.url.path, out)))\r\n\r\n2015-10-21 17:48:25,361: radical.enmd.simulation_analysis_loop.static.default: MainProcess                     : Thread-3       : ERROR   : ComputeUnit error: STDERR: Traceback (most recent call last):\r\n  File \"selection.py\", line 5, in <module>\r\n    from lsdmap.rw import reader\r\nImportError: No module named lsdmap.rw\r\nTraceback (most recent call last):\r\n  File \"reweighting.py\", line 8, in <module>\r\n    from lsdmap.rw import reader\r\nImportError: No module named lsdmap.rw\r\nTraceback (most recent call last):\r\n  File \"spliter.py\", line 27, in <module>\r\n    grofile_obj = gro.GroFile(os.path.dirname(os.path.abspath(__file__)) + '/' + grofile_name)\r\n  File \"/fs4/y14/y14/guest02/radical.pilot.sandbox/rp.session.workflow.iu.xsede.org.ibethune.016729.0021-pilot.0000/unit.000000/gro.py\", line 11, in __init__\r\n    self.natoms=self.get_natoms()\r\n  File \"/fs4/y14/y14/guest02/radical.pilot.sandbox/rp.session.workflow.iu.xsede.org.ibethune.016729.0021-pilot.0000/unit.000000/gro.py\", line 17, in get_natoms\r\n    natoms=int(linecache.getline(self.filename, 2))\r\nValueError: invalid literal for int() with base 10: ''\r\n, STDOUT: Traceback (most recent call last):\r\n  File \"selection.py\", line 5, in <module>\r\n    from lsdmap.rw import reader\r\nImportError: No module named lsdmap.rw\r\nTraceback (most recent call last):\r\n  File \"reweighting.py\", line 8, in <module>\r\n    from lsdmap.rw import reader\r\nImportError: No module named lsdmap.rw\r\nTraceback (most recent call last):\r\n  File \"spliter.py\", line 27, in <module>\r\n    grofile_obj = gro.GroFile(os.path.dirname(os.path.abspath(__file__)) + '/' + grofile_name)\r\n  File \"/fs4/y14/y14/guest02/radical.pilot.sandbox/rp.session.workflow.iu.xsede.org.ibethune.016729.0021-pilot.0000/unit.000000/gro.py\", line 11, in __init__\r\n    self.natoms=self.get_natoms()\r\n  File \"/fs4/y14/y14/guest02/radical.pilot.sandbox/rp.session.workflow.iu.xsede.org.ibethune.016729.0021-pilot.0000/unit.000000/gro.py\", line 17, in get_natoms\r\n    natoms=int(linecache.getline(self.filename, 2))\r\nValueError: invalid literal for int() with base 10: ''\r\n\r\n2015-10-21 17:48:25,361: radical.enmd.simulation_analysis_loop.static.default: MainProcess                     : Thread-3       : ERROR   : Pattern execution FAILED.\r\n2015-10-21 17:48:25,362: radical.pilot       : MainProcess                     : Thread-3       : ERROR   : unit manager controller thread caught system exit -- forcing application shutdown\r\nTraceback (most recent call last):\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/radical/pilot/controller/unit_manager_controller.py\", line 261, in run\r\n    self.call_unit_state_callbacks(unit_id, new_state)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/radical/pilot/controller/unit_manager_controller.py\", line 198, in call_unit_state_callbacks\r\n    cb(self._shared_data[unit_id]['facade_object'], new_state)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/radical/ensemblemd/exec_plugins/simulation_analysis_loop/static.py\", line 141, in unit_state_cb\r\n    sys.exit(1)\r\nSystemExit: 1\r\nExecution interuptedTraceback (most recent call last):\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/radical/ensemblemd/exec_plugins/simulation_analysis_loop/static.py\", line 752, in execute_pattern\r\n    failed_units += \" * Analysis task {0} failed with an error: {1}\\n\".format(unit.uid, unit.stderr)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/radical/pilot/compute_unit.py\", line 261, in stderr\r\n    return self._worker.get_compute_unit_stderr(self.uid)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/radical/pilot/controller/unit_manager_controller.py\", line 390, in get_compute_unit_stderr\r\n    return self._dbs.get_compute_unit_stderr(compute_unit_uid)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/radical/pilot/db/database.py\", line 219, in get_compute_unit_stderr\r\n    return cursor[0]['stderr']\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/pymongo/cursor.py\", line 595, in __getitem__\r\n    for doc in clone:\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/pymongo/cursor.py\", line 1076, in next\r\n    if len(self.__data) or self._refresh():\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/pymongo/cursor.py\", line 1020, in _refresh\r\n    self.__uuid_subtype))\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/pymongo/cursor.py\", line 933, in __send_message\r\n    res = client._send_message_with_response(message, **kwargs)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/pymongo/mongo_client.py\", line 1205, in _send_message_with_response\r\n    response = self.__send_and_receive(message, sock_info)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/pymongo/mongo_client.py\", line 1182, in __send_and_receive\r\n    return self.__receive_message_on_socket(1, request_id, sock_info)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/pymongo/mongo_client.py\", line 1165, in __receive_message_on_socket\r\n    header = self.__receive_data_on_socket(16, sock_info)\r\n  File \"/home/ibethune/extasy-test/lib/python2.7/site-packages/pymongo/mongo_client.py\", line 1153, in __receive_data_on_socket\r\n    chunk = sock_info.sock.recv(length)\r\nKeyboardInterrupt\r\n\r\n        Starting Deallocation                                               done \r\n```\r\n\r\nThe CU with the problem is the lsdmap kernel, and the STDERR there contains information which is relevant:\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"selection.py\", line 5, in <module>\r\n    from lsdmap.rw import reader\r\nImportError: No module named lsdmap.rw\r\nTraceback (most recent call last):\r\n  File \"reweighting.py\", line 8, in <module>\r\n    from lsdmap.rw import reader\r\nImportError: No module named lsdmap.rw\r\nTraceback (most recent call last):\r\n  File \"spliter.py\", line 27, in <module>\r\n    grofile_obj = gro.GroFile(os.path.dirname(os.path.abspath(__file__)) + '/' + grofile_name)\r\n  File \"/fs4/y14/y14/guest02/radical.pilot.sandbox/rp.session.workflow.iu.xsede.org.ibethune.016729.0021-pilot.0000/unit.000000/gro.py\", line 11, in __init__\r\n    self.natoms=self.get_natoms()\r\n  File \"/fs4/y14/y14/guest02/radical.pilot.sandbox/rp.session.workflow.iu.xsede.org.ibethune.016729.0021-pilot.0000/unit.000000/gro.py\", line 17, in get_natoms\r\n    natoms=int(linecache.getline(self.filename, 2))\r\nValueError: invalid literal for int() with base 10: ''\r\n```\r\n\r\nThere seems to be something wrong with the lsdmap installation on ARCHER (maybe?).  If someone can advise we can try to get it fixed tomorrow morning!"}}