{"action":"created","issue":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/413","labels_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/413/labels{/name}","comments_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/413/comments","events_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/413/events","html_url":"https://github.com/Lasagne/Lasagne/issues/413","id":106396749,"number":413,"title":"Latest update GPU compilation errors (No valid section marker was found in file error)","user":{"login":"JesseBuesking","id":1291657,"avatar_url":"https://avatars.githubusercontent.com/u/1291657?v=3","gravatar_id":"","url":"https://api.github.com/users/JesseBuesking","html_url":"https://github.com/JesseBuesking","followers_url":"https://api.github.com/users/JesseBuesking/followers","following_url":"https://api.github.com/users/JesseBuesking/following{/other_user}","gists_url":"https://api.github.com/users/JesseBuesking/gists{/gist_id}","starred_url":"https://api.github.com/users/JesseBuesking/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JesseBuesking/subscriptions","organizations_url":"https://api.github.com/users/JesseBuesking/orgs","repos_url":"https://api.github.com/users/JesseBuesking/repos","events_url":"https://api.github.com/users/JesseBuesking/events{/privacy}","received_events_url":"https://api.github.com/users/JesseBuesking/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2015-09-14T18:04:15Z","updated_at":"2015-09-18T20:06:58Z","closed_at":null,"body":"I updated Lasagne (and in turn Theano) in order to get access to ``lasagne.random`` but in doing so it appears there's an issue, which may or may not be with Theano.\r\n\r\nI followed the [installation instructions](http://lasagne.readthedocs.org/en/latest/user/installation.html#stable-lasagne-release), but now when running the Mnist example I receive a bunch of compilation errors, like the following:\r\n\r\n```\r\nUsing gpu device 0: GeForce GTX 750 Ti (CNMeM is disabled)\r\nLoading data...\r\nBuilding model and compiling functions...\r\nERROR (theano.gof.opt): Optimization failure due to: LocalOptGroup(local_conv_dnn,local_conv_gemm)\r\nERROR (theano.gof.opt): TRACEBACK:\r\nERROR (theano.gof.opt): Traceback (most recent call last):\r\n  File \"/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py\", line 1484, in process_node\r\n    replacements = lopt.transform(node)\r\n  File \"/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py\", line 994, in transform\r\n    repl = opt.transform(node)\r\n  File \"/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/dnn.py\", line 2035, in local_conv_dnn\r\n    direction_hint=direction_hint)\r\n  File \"/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/dnn.py\", line 1197, in dnn_conv\r\n    return GpuDnnConv(algo=algo)(img, kerns, out, desc)\r\nERROR (theano.gof.opt): Optimization failure due to: LocalOptGroup(local_conv_dnn,local_conv_gemm)\r\nERROR (theano.gof.opt): TRACEBACK:\r\nERROR (theano.gof.opt): Traceback (most recent call last):\r\n  File \"/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py\", line 1484, in process_node\r\n    replacements = lopt.transform(node)\r\n  File \"/usr/local/lib/python2.7/dist-packages/theano/gof/opt.py\", line 994, in transform\r\n    repl = opt.transform(node)\r\n  File \"/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/dnn.py\", line 2035, in local_conv_dnn\r\n    direction_hint=direction_hint)\r\n  File \"/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/dnn.py\", line 1197, in dnn_conv\r\n    return GpuDnnConv(algo=algo)(img, kerns, out, desc)\r\n  File \"/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/dnn.py\", line 431, in __init__\r\n    \"APPLY_SPECIFIC(conv_fwd)\")\r\n  File \"/usr/local/lib/python2.7/dist-packages/theano/gof/op.py\", line 1099, in __init__\r\n    self.load_c_code()\r\n  File \"/usr/local/lib/python2.7/dist-packages/theano/gof/op.py\", line 1174, in load_c_code\r\n    \"%s\" % self.func_files[i])\r\nValueError: No valid section marker was found in file /usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/dnn_base.c\r\n...\r\n```\r\n\r\nThe errors continue, each of which appears to result in the same ```ValueError: No valid section marker was found in file``` error.\r\n\r\nI've tried reverting the Theano version, but then I see a previous issue (https://github.com/Lasagne/Lasagne/issues/339).\r\n\r\nI also attempted to update cuda itself from 7.0 to 7.5, but the issue persists.\r\n\r\nGPU code does continue to work, however the Mnist network I was running before which took ~8s per epoch now takes ~30s per epoch, so it appears that GPU optimizations are failing (as noted by the error above ```ERROR (theano.gof.opt): Optimization failure due to...```).\r\n\r\nAdditional info:\r\n```\r\n>> nvcc --version\r\nnvcc: NVIDIA (R) Cuda compiler driver\r\nCopyright (c) 2005-2015 NVIDIA Corporation\r\nBuilt on Tue_Aug_11_14:27:32_CDT_2015\r\nCuda compilation tools, release 7.5, V7.5.17\r\n>> python -c \"import theano; print(theano.__version__)\"\r\nUsing gpu device 0: GeForce GTX 750 Ti (CNMeM is disabled)\r\n0.7.0.dev-15c90dd36c95ea2989a2c0085399147d15a6a477\r\n>> python -c \"import lasagne; print(lasagne.__version__)\"\r\nUsing gpu device 0: GeForce GTX 750 Ti (CNMeM is disabled)\r\n0.1\r\n>> python -c \"import theano; print theano.sandbox.cuda.dnn.dnn_available() or theano.sandbox.cuda.dnn.dnn_available.msg\"\r\nUsing gpu device 0: GeForce GTX 750 Ti (CNMeM is disabled)\r\nTrue\r\n```\r\n\r\nPlease let me know if there's any additional information needed."},"comment":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/comments/141553415","html_url":"https://github.com/Lasagne/Lasagne/issues/413#issuecomment-141553415","issue_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/413","id":141553415,"user":{"login":"JesseBuesking","id":1291657,"avatar_url":"https://avatars.githubusercontent.com/u/1291657?v=3","gravatar_id":"","url":"https://api.github.com/users/JesseBuesking","html_url":"https://github.com/JesseBuesking","followers_url":"https://api.github.com/users/JesseBuesking/followers","following_url":"https://api.github.com/users/JesseBuesking/following{/other_user}","gists_url":"https://api.github.com/users/JesseBuesking/gists{/gist_id}","starred_url":"https://api.github.com/users/JesseBuesking/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JesseBuesking/subscriptions","organizations_url":"https://api.github.com/users/JesseBuesking/orgs","repos_url":"https://api.github.com/users/JesseBuesking/repos","events_url":"https://api.github.com/users/JesseBuesking/events{/privacy}","received_events_url":"https://api.github.com/users/JesseBuesking/received_events","type":"User","site_admin":false},"created_at":"2015-09-18T20:06:58Z","updated_at":"2015-09-18T20:06:58Z","body":"@benanne good to know, thanks for the tip!\r\n\r\n```\r\npartial_sum   mem      min\r\n0             15.8%    1:25\r\n1             17.2%    1:06\r\n```"}}