{"action":"created","issue":{"url":"https://api.github.com/repos/cBio/cbio-cluster/issues/293","labels_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/293/labels{/name}","comments_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/293/comments","events_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/293/events","html_url":"https://github.com/cBio/cbio-cluster/issues/293","id":98023368,"number":293,"title":"Torch CUDA package does not work on gpu-1-5 and gpu-2-7","user":{"login":"LijieTu","id":10173754,"avatar_url":"https://avatars.githubusercontent.com/u/10173754?v=3","gravatar_id":"","url":"https://api.github.com/users/LijieTu","html_url":"https://github.com/LijieTu","followers_url":"https://api.github.com/users/LijieTu/followers","following_url":"https://api.github.com/users/LijieTu/following{/other_user}","gists_url":"https://api.github.com/users/LijieTu/gists{/gist_id}","starred_url":"https://api.github.com/users/LijieTu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LijieTu/subscriptions","organizations_url":"https://api.github.com/users/LijieTu/orgs","repos_url":"https://api.github.com/users/LijieTu/repos","events_url":"https://api.github.com/users/LijieTu/events{/privacy}","received_events_url":"https://api.github.com/users/LijieTu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":70,"created_at":"2015-07-29T20:33:19Z","updated_at":"2015-07-29T23:27:22Z","closed_at":null,"body":"I try running torch CUDA package on some nodes, but something weird happens\r\n\r\n1. First reserve a node:\r\n\r\n`qsub -I -l nodes=gpu-1-5:ppn=1,walltime=02:00:00,mem=2gb,vmem=2gb,pmem=2gb,gpus=4 -N QLOGIN -q gpu`\r\n\r\n2. Then run the docker image: `kaixhin/cuda-torch`\r\n`docker run -it --device /dev/nvidiactl:/dev/nvidiactl --device /dev/nvidia-uvm:/dev/nvidia-uvm --device /dev/nvidia0:/dev/nvidia0 --device /dev/nvidia1:/dev/nvidia1 --device /dev/nvidia2:/dev/nvidia2 --device /dev/nvidia3:/dev/nvidia3 kaixhin/cuda-torch bash`\r\n\r\n3.Type in `\"qlua\"` or `\"th\"`\r\n4. `require 'torch'; require 'cutorch'`\r\n\r\nFor `gpu-1-5` and `gpu-2-7`, I got message like:\r\ncuda runtime error (10) : invalid device ordinal at /root/torch/extra/cutorch/lib/THC/THCTensorRandom.cu:20\r\n\r\nI checked the CUDA, it should be fine.\r\nlijie@gpu-1-5 ~ > `echo $CUDA_VISIBLE_DEVICES`\r\n`3,2,1,0` \r\n\r\nThis situation only happens on these two nodes, I tried the same on gpu-2-5, no errors."},"comment":{"url":"https://api.github.com/repos/cBio/cbio-cluster/issues/comments/126126369","html_url":"https://github.com/cBio/cbio-cluster/issues/293#issuecomment-126126369","issue_url":"https://api.github.com/repos/cBio/cbio-cluster/issues/293","id":126126369,"user":{"login":"LijieTu","id":10173754,"avatar_url":"https://avatars.githubusercontent.com/u/10173754?v=3","gravatar_id":"","url":"https://api.github.com/users/LijieTu","html_url":"https://github.com/LijieTu","followers_url":"https://api.github.com/users/LijieTu/followers","following_url":"https://api.github.com/users/LijieTu/following{/other_user}","gists_url":"https://api.github.com/users/LijieTu/gists{/gist_id}","starred_url":"https://api.github.com/users/LijieTu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LijieTu/subscriptions","organizations_url":"https://api.github.com/users/LijieTu/orgs","repos_url":"https://api.github.com/users/LijieTu/repos","events_url":"https://api.github.com/users/LijieTu/events{/privacy}","received_events_url":"https://api.github.com/users/LijieTu/received_events","type":"User","site_admin":false},"created_at":"2015-07-29T23:27:22Z","updated_at":"2015-07-29T23:27:22Z","body":"I'm on `GPU-1-7`\r\n\r\n```\r\nlijie@mskcc-ln1 ~ > qsub -I -l nodes=1:ppn=1:gpus=4:shared -l walltime=02:00:00,mem=2gb,vmem=2gb,pmem=2gb -N QLOGIN -q gpu\r\nqsub: waiting for job 4692471.mskcc-fe1.local to start\r\nqsub: job 4692471.mskcc-fe1.local ready\r\n\r\nlijie@gpu-1-7 ~ > nvidia-smi\r\nWed Jul 29 19:18:44 2015       \r\n+------------------------------------------------------+                       \r\n| NVIDIA-SMI 346.46     Driver Version: 346.46         |                       \r\n|-------------------------------+----------------------+----------------------+\r\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n|===============================+======================+======================|\r\n|   0  GeForce GTX 680     Off  | 0000:03:00.0     N/A |                  N/A |\r\n| 30%   31C    P8    N/A /  N/A |     10MiB /  4095MiB |     N/A      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   1  GeForce GTX 680     Off  | 0000:04:00.0     N/A |                  N/A |\r\n| 30%   31C    P8    N/A /  N/A |     10MiB /  4095MiB |     N/A      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   2  GeForce GTX 680     Off  | 0000:83:00.0     N/A |                  N/A |\r\n| 30%   29C    P8    N/A /  N/A |     10MiB /  4095MiB |     N/A      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n|   3  GeForce GTX 680     Off  | 0000:84:00.0     N/A |                  N/A |\r\n| 30%   30C    P8    N/A /  N/A |     10MiB /  4095MiB |     N/A      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n                                                                               \r\n+-----------------------------------------------------------------------------+\r\n| Processes:                                                       GPU Memory |\r\n|  GPU       PID  Type  Process name                               Usage      |\r\n|=============================================================================|\r\n|    0            C+G   Not Supported                                         |\r\n|    1            C+G   Not Supported                                         |\r\n|    2            C+G   Not Supported                                         |\r\n|    3            C+G   Not Supported                                         |\r\n+-----------------------------------------------------------------------------+\r\nlijie@gpu-1-7 ~ > docker run -it --device /dev/nvidiactl:/dev/nvidiactl --device /dev/nvidia-uvm:/dev/nvidia-uvm --device /dev/nvidia0:/dev/nvidia0 --device /dev/nvidia1:/dev/nvidia1 --device /dev/nvidia2:/dev/nvidia2 --device /dev/nvidia3:/dev/nvidia3 kaixhin/cuda-torch bash\r\nroot@cc1738761e8f:~/torch# qlua\r\nUnable to connect X11 server (continuing with -nographics)\r\nLua 5.1  Copyright (C) 1994-2008 Lua.org, PUC-Rio\r\n> require 'torch'\r\n> require 'cutorch'\r\n\r\n```\r\n"}}