{"action":"created","issue":{"url":"https://api.github.com/repos/cBio/cbio-cluster/issues/258","labels_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/258/labels{/name}","comments_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/258/comments","events_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/258/events","html_url":"https://github.com/cBio/cbio-cluster/issues/258","id":77394665,"number":258,"title":"CUDA+OpenMM problems","user":{"login":"danielparton","id":4217946,"avatar_url":"https://avatars.githubusercontent.com/u/4217946?v=3","gravatar_id":"","url":"https://api.github.com/users/danielparton","html_url":"https://github.com/danielparton","followers_url":"https://api.github.com/users/danielparton/followers","following_url":"https://api.github.com/users/danielparton/following{/other_user}","gists_url":"https://api.github.com/users/danielparton/gists{/gist_id}","starred_url":"https://api.github.com/users/danielparton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielparton/subscriptions","organizations_url":"https://api.github.com/users/danielparton/orgs","repos_url":"https://api.github.com/users/danielparton/repos","events_url":"https://api.github.com/users/danielparton/events{/privacy}","received_events_url":"https://api.github.com/users/danielparton/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":25,"created_at":"2015-05-17T20:45:33Z","updated_at":"2015-05-18T00:55:47Z","closed_at":null,"body":"I'm having trouble getting some OpenMM simulations to run. My guess is that it is something to do with the cluster environment or the way I am submitting the jobs (rather than a software problem), but I'm really not sure at this point.\r\n\r\nI've been testing with single-GPU jobs, using the following qsub options: `-q gpu -l procs=1,gpus=1:shared`\r\nEach job performs a series of consecutive OpenMM simulations, each of which normally lasts around 5 minutes.\r\nThe typical behavior is for ~6-10 simulations to complete successfully, then one of two things happen:\r\n1. the next simulation completes a few iterations, then simply hangs\r\n2. the next simulation completes a few iterations, then returns an exception as follows:\r\n```\r\nFile \"/cbio/jclab/home/parton/opt/anaconda/lib/python2.7/site-packages/simtk/openmm/openmm.py\", line 11499, in step\r\n    return _openmm.LangevinIntegrator_step(self, *args)\r\nException: Error downloading array posq: Invalid error code (700)\r\n```\r\n\r\nIn the second case, my Python wrapper program continues to execute, since the OpenMM call is wrapped with an exception-handler. All subsequent simulations then fail upon initialization with the following exception message:\r\n\r\n```\r\nFile \"/cbio/jclab/home/parton/opt/anaconda/lib/python2.7/site-packages/simtk/openmm/openmm.py\", line 4879, in __init__\r\n    this = _openmm.new_Context(*args)\r\nException: Error initializing Context: CUDA_ERROR_INVALID_DEVICE (101) at /cbio/jclab/home/parton/opt/openmm/openmm/platforms/cuda/src/CudaContext.cpp:149\r\n```\r\n\r\nIn both cases I have been able to complete the same simulation on a second attempt, using the same GPU. If the program is left to run, then the same problem occurs a few simulations later.\r\n\r\nAlso, these errors only seem to occur with explicit-solvent MD simulations. My implicit-solvent MD simulations (which also use the OpenMM CUDA implementation) have been running without any problems.\r\n\r\nAny suggestions on what might be the issue, or how I could debug this further?\r\n\r\n### Environment details\r\n* OpenMM version - 6.1; commit 751ff07958\r\n* CUDA version - 6.5\r\n* mpirun version: HYDRA 1.4.1p1 (from Anaconda)\r\n* GPU ids assigned using the CUDA_VISIBLE_DEVICES environment variable, set using the script [`build_mpirun_configfile`](https://github.com/choderalab/clusterutils/blob/master/clusterutils/build_mpirun_configfile.py)"},"comment":{"url":"https://api.github.com/repos/cBio/cbio-cluster/issues/comments/102880084","html_url":"https://github.com/cBio/cbio-cluster/issues/258#issuecomment-102880084","issue_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/258","id":102880084,"user":{"login":"danielparton","id":4217946,"avatar_url":"https://avatars.githubusercontent.com/u/4217946?v=3","gravatar_id":"","url":"https://api.github.com/users/danielparton","html_url":"https://github.com/danielparton","followers_url":"https://api.github.com/users/danielparton/followers","following_url":"https://api.github.com/users/danielparton/following{/other_user}","gists_url":"https://api.github.com/users/danielparton/gists{/gist_id}","starred_url":"https://api.github.com/users/danielparton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielparton/subscriptions","organizations_url":"https://api.github.com/users/danielparton/orgs","repos_url":"https://api.github.com/users/danielparton/repos","events_url":"https://api.github.com/users/danielparton/events{/privacy}","received_events_url":"https://api.github.com/users/danielparton/received_events","type":"User","site_admin":false},"created_at":"2015-05-18T00:55:47Z","updated_at":"2015-05-18T00:55:47Z","body":"Ok I've tried this out again with OpenMM 6.2. So far it has been the same behavior (with error message, rather than the \"hang-with-no-output\" behavior).\r\n\r\n5th simulation:\r\n```\r\n  File \"/cbio/jclab/home/parton/opt/anaconda/lib/python2.7/site-packages/simtk/openmm/openmm.py\", line 12102, in step\r\n    return _openmm.LangevinIntegrator_step(self, *args)\r\nException: Error downloading array energyBuffer: Invalid error code (700)\r\n```\r\n6th and subsequent simulations:\r\n```\r\n  File \"/cbio/jclab/home/parton/opt/anaconda/lib/python2.7/site-packages/simtk/openmm/openmm.py\", line 15302, in __init__\r\n    this = _openmm.new_Context(*args)\r\nException: Error initializing Context: CUDA_ERROR_INVALID_DEVICE (101) at /cbio/jclab/home/parton/opt/openmm/openmm/platforms/cuda/src/CudaContext.cpp:149\r\n```\r\n\r\nHost details:\r\n* gpu-2-17 // GPU 0"}}