{"action":"opened","issue":{"url":"https://api.github.com/repos/cBio/cbio-cluster/issues/275","labels_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/275/labels{/name}","comments_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/275/comments","events_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/275/events","html_url":"https://github.com/cBio/cbio-cluster/issues/275","id":89946728,"number":275,"title":"GPUs are not set to appropriate thread mode by Torque","user":{"login":"jchodera","id":3656088,"avatar_url":"https://avatars.githubusercontent.com/u/3656088?v=3","gravatar_id":"","url":"https://api.github.com/users/jchodera","html_url":"https://github.com/jchodera","followers_url":"https://api.github.com/users/jchodera/followers","following_url":"https://api.github.com/users/jchodera/following{/other_user}","gists_url":"https://api.github.com/users/jchodera/gists{/gist_id}","starred_url":"https://api.github.com/users/jchodera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchodera/subscriptions","organizations_url":"https://api.github.com/users/jchodera/orgs","repos_url":"https://api.github.com/users/jchodera/repos","events_url":"https://api.github.com/users/jchodera/events{/privacy}","received_events_url":"https://api.github.com/users/jchodera/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-06-21T18:59:07Z","updated_at":"2015-06-21T18:59:07Z","closed_at":null,"body":"Here's another Torque bug:\r\n\r\nWhen I request GPU processes across any nodes with\r\n```\r\n#PBS -l procs=8,gpus=1:shared\r\n```\r\nthe GPU mode (here, `shared`) is not being set correctly on the GPUs allocated.\r\n\r\nFrom `$PBS_GPUFILE` (for job `3671532`), we have:\r\n```\r\ngpu-1-11-gpu3\r\ngpu-1-11-gpu2\r\ngpu-1-11-gpu1\r\ngpu-1-11-gpu0\r\ngpu-2-9-gpu3\r\ngpu-2-9-gpu2\r\ngpu-2-9-gpu1\r\ngpu-2-9-gpu0\r\n```\r\nbut if you go to `gpu-1-11` and run `nvidia-smi`, you see that the GPUs are still in thread-exclusive mode:\r\n```\r\n[chodera@gpu-1-11 ~]$ nvidia-smi\r\nSun Jun 21 14:56:54 2015       \r\n+------------------------------------------------------+                       \r\n| NVIDIA-SMI 346.46     Driver Version: 346.46         |                       \r\n|-------------------------------+----------------------+----------------------+\r\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n|===============================+======================+======================|\r\n|   0  GeForce GTX 680     Off  | 0000:03:00.0     N/A |                  N/A |\r\n| 30%   38C    P0    N/A /  N/A |     62MiB /  4095MiB |     N/A    E. Thread |\r\n+-------------------------------+----------------------+----------------------+\r\n|   1  GeForce GTX 680     Off  | 0000:04:00.0     N/A |                  N/A |\r\n| 30%   36C    P0    N/A /  N/A |     62MiB /  4095MiB |     N/A    E. Thread |\r\n+-------------------------------+----------------------+----------------------+\r\n|   2  GeForce GTX 680     Off  | 0000:83:00.0     N/A |                  N/A |\r\n| 30%   35C    P0    N/A /  N/A |     62MiB /  4095MiB |     N/A    E. Thread |\r\n+-------------------------------+----------------------+----------------------+\r\n|   3  GeForce GTX 680     Off  | 0000:84:00.0     N/A |                  N/A |\r\n| 30%   38C    P0    N/A /  N/A |     62MiB /  4095MiB |     N/A    E. Thread |\r\n+-------------------------------+----------------------+----------------------+\r\n                                                                               \r\n+-----------------------------------------------------------------------------+\r\n| Processes:                                                       GPU Memory |\r\n|  GPU       PID  Type  Process name                               Usage      |\r\n|=============================================================================|\r\n|    0            C+G   Not Supported                                         |\r\n|    1            C+G   Not Supported                                         |\r\n|    2            C+G   Not Supported                                         |\r\n|    3            C+G   Not Supported                                         |\r\n+-----------------------------------------------------------------------------+\r\n```"}}