{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/601","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/601/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/601/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/601/events","html_url":"https://github.com/tensorflow/tensorflow/issues/601","id":123702970,"number":601,"title":"failed call to cuInit: CUDA_ERROR_UNKNOWN after Docker build on Macbook Pro (Late 2013) with Linux","user":{"login":"dcunited001","id":782627,"avatar_url":"https://avatars.githubusercontent.com/u/782627?v=3","gravatar_id":"","url":"https://api.github.com/users/dcunited001","html_url":"https://github.com/dcunited001","followers_url":"https://api.github.com/users/dcunited001/followers","following_url":"https://api.github.com/users/dcunited001/following{/other_user}","gists_url":"https://api.github.com/users/dcunited001/gists{/gist_id}","starred_url":"https://api.github.com/users/dcunited001/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dcunited001/subscriptions","organizations_url":"https://api.github.com/users/dcunited001/orgs","repos_url":"https://api.github.com/users/dcunited001/repos","events_url":"https://api.github.com/users/dcunited001/events{/privacy}","received_events_url":"https://api.github.com/users/dcunited001/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-12-23T18:18:08Z","updated_at":"2015-12-28T01:58:19Z","closed_at":null,"body":"This issue is similar to to #394. I believe i'm seeing it because tensorflow can't find `libcuda.so`.\r\n\r\nI know that my `libcuda.so` can be found in `/usr/lib/x86_64-linux-gnu/`, but is this accessible to the compiler and to python without further configuration?  I tried adding it to my docker user's `$LD_LIBRARY_PATH`, but that did not fix the problem. \r\n\r\nI'm running on:\r\n- Macbook Pro (Late 2013) Hardware\r\n- Mint 17\r\n- 3.0 Compute Capability\r\n- CUDA 7.0 & cuDNN 2.0 \r\n- running docker service under a docker user & group.  no sudo access for docker user.\r\n\r\non host os, values for `$CUDA_SO` and `$DEVICES` that are passed into `./docker_run_gpu.sh` are:\r\n\r\n```\r\n$ export CUDA_SO=$(\\ls /usr/lib/x86_64-linux-gnu/libcuda* | xargs -I{} echo '-v {}:{}')\r\n$ export DEVICES=$(\\ls /dev/nvidia* | xargs -I{} echo '--device {}:{}')\r\n\r\n$ echo $CUDA_SO\r\n-v /usr/lib/x86_64-linux-gnu/libcuda.so:/usr/lib/x86_64-linux-gnu/libcuda.so -v /usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1 -v /usr/lib/x86_64-linux-gnu/libcuda.so.352.68:/usr/lib/x86_64-linux-gnu/libcuda.so.352.68\r\n\r\n$ echo $DEVICES\r\n--device /dev/nvidia0:/dev/nvidia0 --device /dev/nvidiactl:/dev/nvidiactl\r\n```\r\n\r\nI customized `Dockerfile.devel-gpu` as below.  I prepended `TF_CUDA_COMPUTE_CAPABILITIES=3.0` and  `TF_UNOFFICIAL_SETTING=1` for the `./configure` step.  \r\n\r\nShould I add the `libcuda.so` path to `ENV LD_LIBRARY_PATH`?\r\n\r\n```Dockerfile\r\n# ........\r\n\r\nRUN git clone --recursive https://github.com/tensorflow/tensorflow.git && \\\r\n    cd tensorflow && \\\r\n    git checkout 0.6.0\r\nWORKDIR /tensorflow\r\n\r\n# Configure the build for our CUDA configuration.\r\nENV CUDA_TOOLKIT_PATH /usr/local/cuda\r\nENV CUDNN_INSTALL_PATH /usr/local/cuda\r\nENV TF_NEED_CUDA 1\r\n\r\nRUN TF_CUDA_COMPUTE_CAPABILITIES=3.0 TF_UNOFFICIAL_SETTING=1 ./configure && \\\r\n    bazel build -c opt --config=cuda tensorflow/tools/pip_package:build_pip_package && \\\r\n    bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/pip && \\\r\n    pip install --upgrade /tmp/pip/tensorflow-*.whl\r\n\r\nWORKDIR /root\r\n\r\n# Set up CUDA variables\r\nENV CUDA_PATH /usr/local/cuda\r\nENV LD_LIBRARY_PATH /usr/local/cuda/lib64\r\n\r\n# TensorBoard\r\nEXPOSE 6006\r\n# IPython\r\nEXPOSE 8888\r\n\r\nRUN [\"/bin/bash\"]\r\n```\r\n\r\nThen I built the image and ran it with `./docker_run_gpu.sh tf/tf`\r\n\r\nStarted tensorflow with: `python -m tensorflow.models.image.mnist.convolutional`\r\n\r\nAnd I'm getting `failed call to cuInit: CUDA_ERROR_UNKNOWN` when tensorflow starts up. But this is after reporting that it `successfully opened CUDA library libcuda.so locally`.\r\n\r\n```\r\nI tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcublas.so.7.0 locally\r\nI tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcudnn.so.6.5 locally\r\nI tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcufft.so.7.0 locally\r\nI tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcuda.so locally\r\nI tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcurand.so.7.0 locally\r\n\r\nI tensorflow/core/common_runtime/local_device.cc:40] Local device intra op parallelism threads: 8\r\nmodprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/3.19.0-32-generic/modules.dep.bin'\r\nE tensorflow/stream_executor/cuda/cuda_driver.cc:481] failed call to cuInit: CUDA_ERROR_UNKNOWN\r\nI tensorflow/stream_executor/cuda/cuda_diagnostics.cc:114] retrieving CUDA diagnostic information for host: 24b008aee65f\r\nI tensorflow/stream_executor/cuda/cuda_diagnostics.cc:121] hostname: 24b008aee65f\r\nI tensorflow/stream_executor/cuda/cuda_diagnostics.cc:146] libcuda reported version is: Not found: was unable to find libcuda.so DSO loaded into this program\r\nI tensorflow/stream_executor/cuda/cuda_diagnostics.cc:257] driver version file contents: \"\"\"NVRM version: NVIDIA UNIX x86_64 Kernel Module  352.68  Tue Dec  1 17:24:11 PST 2015\r\nGCC version:  gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/167456711","html_url":"https://github.com/tensorflow/tensorflow/issues/601#issuecomment-167456711","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/601","id":167456711,"user":{"login":"ebrevdo","id":1794715,"avatar_url":"https://avatars.githubusercontent.com/u/1794715?v=3","gravatar_id":"","url":"https://api.github.com/users/ebrevdo","html_url":"https://github.com/ebrevdo","followers_url":"https://api.github.com/users/ebrevdo/followers","following_url":"https://api.github.com/users/ebrevdo/following{/other_user}","gists_url":"https://api.github.com/users/ebrevdo/gists{/gist_id}","starred_url":"https://api.github.com/users/ebrevdo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ebrevdo/subscriptions","organizations_url":"https://api.github.com/users/ebrevdo/orgs","repos_url":"https://api.github.com/users/ebrevdo/repos","events_url":"https://api.github.com/users/ebrevdo/events{/privacy}","received_events_url":"https://api.github.com/users/ebrevdo/received_events","type":"User","site_admin":false},"created_at":"2015-12-28T01:58:19Z","updated_at":"2015-12-28T01:58:19Z","body":"You should not need to run modprobe in the container, only on the host.\nOnce that runs, the /dev/nvidia* devices are accessible and you can launch\nthe container.\n\nOn Sun, Dec 27, 2015 at 3:52 PM, DC <notifications@github.com> wrote:\n\n> i was able to build on my local machine for now. going to try again with\n> docker later, but i think the issue was my host machine configuration. for\n> some reason the docker container is not permitted or cannot access the\n> nvidia drivers.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/tensorflow/tensorflow/issues/601#issuecomment-167449139>\n> .\n>\n"}}