{"action":"created","issue":{"url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/159","labels_url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/159/labels{/name}","comments_url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/159/comments","events_url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/159/events","html_url":"https://github.com/radical-cybertools/ExTASY/issues/159","id":65415342,"number":159,"title":"Tool hangs at shutdown (GROMACS/LSDMap)","user":{"login":"oleweidner","id":2249976,"avatar_url":"https://avatars.githubusercontent.com/u/2249976?v=3","gravatar_id":"","url":"https://api.github.com/users/oleweidner","html_url":"https://github.com/oleweidner","followers_url":"https://api.github.com/users/oleweidner/followers","following_url":"https://api.github.com/users/oleweidner/following{/other_user}","gists_url":"https://api.github.com/users/oleweidner/gists{/gist_id}","starred_url":"https://api.github.com/users/oleweidner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oleweidner/subscriptions","organizations_url":"https://api.github.com/users/oleweidner/orgs","repos_url":"https://api.github.com/users/oleweidner/repos","events_url":"https://api.github.com/users/oleweidner/events{/privacy}","received_events_url":"https://api.github.com/users/oleweidner/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-03-31T10:09:26Z","updated_at":"2015-04-28T07:10:55Z","closed_at":null,"body":"```\r\n[Callback]: ComputeUnit '551a6eff6bf88ba784d3048f' state changed to Done.\r\n[Callback]: ComputeUnit '551a6eff6bf88ba784d30493' state changed to Done.\r\n[Callback]: ComputeUnit '551a6eff6bf88ba784d3048a' state changed to StagingOutput.\r\n[Callback]: ComputeUnit '551a6eff6bf88ba784d30488' state changed to StagingOutput.\r\n[Callback]: ComputeUnit '551a6eff6bf88ba784d3048a' state changed to Done.\r\n[Callback]: ComputeUnit '551a6eff6bf88ba784d30488' state changed to Done.\r\n[Callback]: ComputePilot '551a6e436bf88ba784d30470' state changed to Canceled.\r\n```\r\n\r\n... never terminates after \"Canceled\". Might be a RP problem. Tested with 0.24 from PyPi. "},"comment":{"url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/comments/96954035","html_url":"https://github.com/radical-cybertools/ExTASY/issues/159#issuecomment-96954035","issue_url":"https://api.github.com/repos/radical-cybertools/ExTASY/issues/159","id":96954035,"user":{"login":"oleweidner","id":2249976,"avatar_url":"https://avatars.githubusercontent.com/u/2249976?v=3","gravatar_id":"","url":"https://api.github.com/users/oleweidner","html_url":"https://github.com/oleweidner","followers_url":"https://api.github.com/users/oleweidner/followers","following_url":"https://api.github.com/users/oleweidner/following{/other_user}","gists_url":"https://api.github.com/users/oleweidner/gists{/gist_id}","starred_url":"https://api.github.com/users/oleweidner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oleweidner/subscriptions","organizations_url":"https://api.github.com/users/oleweidner/orgs","repos_url":"https://api.github.com/users/oleweidner/repos","events_url":"https://api.github.com/users/oleweidner/events{/privacy}","received_events_url":"https://api.github.com/users/oleweidner/received_events","type":"User","site_admin":false},"created_at":"2015-04-28T07:10:55Z","updated_at":"2015-04-28T07:10:55Z","body":"Ok, here's the results of my tests:\r\n\r\n**TEST 1**\r\n\r\nRemoved 'ALLOCATION' from the config file, should result in an error. Yep:\r\n\r\n```\r\nExTASY version :  0.1.4-beta-8-gbcbad40\r\nLoading kernel configurations from /private/tmp/test_extasy/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/amber.json\r\nLoading kernel configurations from /private/tmp/test_extasy/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/coco.json\r\nLoading kernel configurations from /private/tmp/test_extasy/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/gromacs.json\r\nLoading kernel configurations from /private/tmp/test_extasy/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/lsdmap.json\r\nLoading kernel configurations from /private/tmp/test_extasy/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/mmpbsa.json\r\nLoading kernel configurations from /private/tmp/test_extasy/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/namd.json\r\nLoading kernel configurations from /private/tmp/test_extasy/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/sleep.json\r\nLoading kernel configurations from /private/tmp/test_extasy/lib/python2.7/site-packages/radical/ensemblemd/mdkernels/configs/test.json\r\nSession UID: rp.session.teahupoo.local.oweidner.016553.0002 \r\nAn error occurred: 'module' object has no attribute 'ALLOCATION'\r\nException triggered, no session created, exiting now...\r\n^C^C^C^C\r\n```\r\nHangs indefinitely - Ctrl+C doesn't work. The former might be an issue of proper error-handling in ExTASY (all exceptions caught and session.close() called?). The latter is a well-known problem with radical-pilot: https://github.com/radical-cybertools/radical.pilot/issues/448\r\n\r\n**TEST 2** \r\n\r\nForced pilot to fail by using 'localhost' but set an allocation (not supported on localhost). This results in an error and subsequent clean shutdown:\r\n\r\n```\r\nPilot pilot.0000 has FAILED. Can't recover.\r\nPilot log:- \r\n{'timestamp': datetime.datetime(2015, 4, 28, 6, 57, 33, 638000), 'message': u'Using bootstrapper /private/tmp/test_extasy/lib/python2.7/site-packages/radical.pilot-0.28-py2.7.egg/radical/pilot/bootstrapper/default_bootstrapper.sh'}\r\n{'timestamp': datetime.datetime(2015, 4, 28, 6, 57, 33, 639000), 'message': u\"Copying bootstrapper 'file://localhost//private/tmp/test_extasy/lib/python2.7/site-packages/radical.pilot-0.28-py2.7.egg/radical/pilot/bootstrapper/default_bootstrapper.sh' to agent sandbox (file://localhost/Users/oweidner/radical.pilot.sandbox/rp.session.teahupoo.local.oweidner.016553.0003-pilot.0000//pilot_bootstrapper.sh).\"}\r\n{'timestamp': datetime.datetime(2015, 4, 28, 6, 57, 34, 423000), 'message': u\"Copying sdist 'file://localhost//private/tmp/test_extasy/lib/python2.7/site-packages/radical.utils-0.28-py2.7.egg/radical/utils/radical.utils-0.28.tar.gz' to sdist sandbox (file://localhost/Users/oweidner/radical.pilot.sandbox/rp.session.teahupoo.local.oweidner.016553.0003-pilot.0000/).\"}\r\n{'timestamp': datetime.datetime(2015, 4, 28, 6, 57, 34, 647000), 'message': u\"Copying sdist 'file://localhost//tmp/test_extasy/lib/python2.7/site-packages/saga_python-0.28-py2.7.egg/saga/saga-python-0.28.tar.gz' to sdist sandbox (file://localhost/Users/oweidner/radical.pilot.sandbox/rp.session.teahupoo.local.oweidner.016553.0003-pilot.0000/).\"}\r\n{'timestamp': datetime.datetime(2015, 4, 28, 6, 57, 34, 870000), 'message': u\"Copying sdist 'file://localhost//private/tmp/test_extasy/lib/python2.7/site-packages/radical.pilot-0.28-py2.7.egg/radical/pilot/controller/..//radical.pilot-0.28.tar.gz' to sdist sandbox (file://localhost/Users/oweidner/radical.pilot.sandbox/rp.session.teahupoo.local.oweidner.016553.0003-pilot.0000/).\"}\r\n{'timestamp': datetime.datetime(2015, 4, 28, 6, 57, 37, 596000), 'message': u'Submitting SAGA job with description: {\\'Queue\\': \\'normal\\', \\'Executable\\': \\'/bin/bash\\', \\'TotalPhysicalMemory\\': None, \\'WorkingDirectory\\': \\'/Users/oweidner/radical.pilot.sandbox/rp.session.teahupoo.local.oweidner.016553.0003-pilot.0000/\\', \\'Project\\': \\'none\\', \\'WallTimeLimit\\': 20, \\'Arguments\\': [\\'-l pilot_bootstrapper.sh\\', \" -b \\'radical.utils-0.28.tar.gz:saga-python-0.28.tar.gz:radical.pilot-0.28.tar.gz\\' -c \\'2\\' -d \\'10\\' -g \\'/Users/oweidner/radical.pilot.sandbox/ve_localhost\\' -j \\'FORK\\' -k \\'MPIEXEC\\' -l \\'FORK\\' -m \\'extasy-db.epcc.ed.ac.uk:27017\\' -n \\'radicalpilot\\' -o \\'SHELL\\' -p \\'pilot.0000\\' -q \\'CONTINUOUS\\' -r \\'20\\' -s \\'rp.session.teahupoo.local.oweidner.016553.0003\\' -t \\'multicore\\' -u \\'create\\' -v \\'debug\\' -a \\'extasy:extasyproject\\'\"], \\'Error\\': \\'agent.err\\', \\'Output\\': \\'agent.out\\', \\'TotalCPUCount\\': 2}'}\r\n{'timestamp': datetime.datetime(2015, 4, 28, 6, 57, 37, 601000), 'message': u\"Pilot launching failed! ('JobDescription.Project' (none) is not supported by adaptor saga.adaptor.shell_job (/tmp/test_extasy/lib/python2.7/site-packages/saga_python-0.28-py2.7.egg/saga/job/service.py +300 (create_job)  :  raise se.BadParameter._log (self._logger, msg)))\"}\r\nPilot STDOUT : \r\nPilot STDERR : \r\nExecution was interrupted\r\nClosing session, exiting now ...\r\n```\r\n\r\n**CONCLUSION**\r\n\r\nI think this ticket can be closed as things seem to terminate properly in the common error cases. Hijacking terminals is a known problem with radical-pilot and will be addressed as part of the \"Refactor-2\" milestone (https://github.com/radical-cybertools/radical.pilot/milestones/MS-Refactor-2) in radical-pilot (Andre might have details w.r.t. the timeline). \r\n\r\n**On a related note**\r\n\r\nDuring the tests above I came to realize that it is not possible to run ExTASY locally (REMOTE_HOST=localhost). This might be a completely irrelevant use-case, but at least for development purposes this might come in handy sometimes? Currently the ExTASY resource config file requires me to set ALLOCATION (otherwise error ``An error occurred: 'module' object has no attribute 'ALLOCATION'``). However, if I set it to a bogus value, I get a different error (``Pilot launching failed! ('JobDescription.Project' (bogus) is not supported by adaptor saga.adaptor.shell_job``). I think this could be easily fixed by making ALLOCATION an optional parameter in the resource.cfg.\r\n"}}