{"action":"created","issue":{"url":"https://api.github.com/repos/cBio/cbio-cluster/issues/299","labels_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/299/labels{/name}","comments_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/299/comments","events_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/299/events","html_url":"https://github.com/cBio/cbio-cluster/issues/299","id":99913128,"number":299,"title":"Stuck or out-of-queue 'qlua' jobs running on gpu-2-10","user":{"login":"jchodera","id":3656088,"avatar_url":"https://avatars.githubusercontent.com/u/3656088?v=3","gravatar_id":"","url":"https://api.github.com/users/jchodera","html_url":"https://github.com/jchodera","followers_url":"https://api.github.com/users/jchodera/followers","following_url":"https://api.github.com/users/jchodera/following{/other_user}","gists_url":"https://api.github.com/users/jchodera/gists{/gist_id}","starred_url":"https://api.github.com/users/jchodera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchodera/subscriptions","organizations_url":"https://api.github.com/users/jchodera/orgs","repos_url":"https://api.github.com/users/jchodera/repos","events_url":"https://api.github.com/users/jchodera/events{/privacy}","received_events_url":"https://api.github.com/users/jchodera/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2015-08-09T16:50:03Z","updated_at":"2015-08-12T13:14:05Z","closed_at":"2015-08-12T13:14:05Z","body":"I am having GPU jobs fail because they are being allocated to GPU slots on which stuck or out-of-queue `qlua` jobs are running.  For example, on `gpu-2-10`, I was allocated `gpu-2-10-gpu0` as part of Torque array job `5217812`, but `qlua` jobs are running on all nodes with no trace of their parent job:\r\n```\r\n[chodera@gpu-2-10 ~]$ nvidia-smi\r\nSun Aug  9 12:39:24 2015       \r\n+------------------------------------------------------+                       \r\n| NVIDIA-SMI 346.46     Driver Version: 346.46         |                       \r\n|-------------------------------+----------------------+----------------------+\r\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n|===============================+======================+======================|\r\n|   0  GeForce GTX TITAN   Off  | 0000:03:00.0     Off |                  N/A |\r\n| 58%   80C    P0   214W / 250W |   1032MiB /  6143MiB |     98%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   1  GeForce GTX TITAN   Off  | 0000:04:00.0     Off |                  N/A |\r\n| 34%   51C    P0    71W / 250W |    406MiB /  6143MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   2  GeForce GTX TITAN   Off  | 0000:83:00.0     Off |                  N/A |\r\n| 35%   53C    P0    72W / 250W |    406MiB /  6143MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   3  GeForce GTX TITAN   Off  | 0000:84:00.0     Off |                  N/A |\r\n| 35%   52C    P0    71W / 250W |    406MiB /  6143MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n                                                                               \r\n+-----------------------------------------------------------------------------+\r\n| Processes:                                                       GPU Memory |\r\n|  GPU       PID  Type  Process name                               Usage      |\r\n|=============================================================================|\r\n|    0     12848    C   qlua                                           920MiB |\r\n|    0     13261    C   .../NVIDIA/Fermi/beta/Core_21.fah/FahCore_21    93MiB |\r\n|    1     12848    C   qlua                                           388MiB |\r\n|    2     12848    C   qlua                                           388MiB |\r\n|    3     12848    C   qlua                                           388MiB |\r\n+-----------------------------------------------------------------------------+\r\n```\r\nThe GPU was in `Default` (shared) mode, and did not have sufficient resources to run my job because of the `qlua` job, so failed.\r\n\r\nWe are going to have to figure out some solution to make sure GPUs are not bound up by stuck or failed processes.  \r\n\r\nA good way to start is probably to insist that any Torque jobs that use GPUs enable the `exclusive` flag, such as\r\n```\r\n#PBS -l nodes=1:ppn=1:gpus=1:exclusive\r\n```\r\nfor a request to use one GPU and one CPU thread.  This will ensure that we can track which GPUs have been allocated and other processes that try to use this GPU will immediately fail.\r\n\r\nTagging some other docker users: @LijieTu @akahles \r\n\r\nOther ideas for how we can ensure proper cleanup of failed GPU-binding docker processes would be welcomed.  The current state of things can't continue in a shared production environment.  I'd prefer not to take the drastic step of banning GPU docker instances completely (ours included) until we can figure this out, but I'm not sure what else would allow us to continue to make use of the shared GPU resources."},"comment":{"url":"https://api.github.com/repos/cBio/cbio-cluster/issues/comments/130296596","html_url":"https://github.com/cBio/cbio-cluster/issues/299#issuecomment-130296596","issue_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/299","id":130296596,"user":{"login":"tatarsky","id":6439722,"avatar_url":"https://avatars.githubusercontent.com/u/6439722?v=3","gravatar_id":"","url":"https://api.github.com/users/tatarsky","html_url":"https://github.com/tatarsky","followers_url":"https://api.github.com/users/tatarsky/followers","following_url":"https://api.github.com/users/tatarsky/following{/other_user}","gists_url":"https://api.github.com/users/tatarsky/gists{/gist_id}","starred_url":"https://api.github.com/users/tatarsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tatarsky/subscriptions","organizations_url":"https://api.github.com/users/tatarsky/orgs","repos_url":"https://api.github.com/users/tatarsky/repos","events_url":"https://api.github.com/users/tatarsky/events{/privacy}","received_events_url":"https://api.github.com/users/tatarsky/received_events","type":"User","site_admin":false},"created_at":"2015-08-12T13:14:04Z","updated_at":"2015-08-12T13:14:04Z","body":"I believe we have handled most items in this and my script continues in manual test mode. "}}