{"action":"created","issue":{"url":"https://api.github.com/repos/Theano/Theano/issues/3252","labels_url":"https://api.github.com/repos/Theano/Theano/issues/3252/labels{/name}","comments_url":"https://api.github.com/repos/Theano/Theano/issues/3252/comments","events_url":"https://api.github.com/repos/Theano/Theano/issues/3252/events","html_url":"https://github.com/Theano/Theano/pull/3252","id":99186013,"number":3252,"title":"Fix metaoptimizer for cudnn","user":{"login":"f0k","id":629706,"avatar_url":"https://avatars.githubusercontent.com/u/629706?v=3","gravatar_id":"","url":"https://api.github.com/users/f0k","html_url":"https://github.com/f0k","followers_url":"https://api.github.com/users/f0k/followers","following_url":"https://api.github.com/users/f0k/following{/other_user}","gists_url":"https://api.github.com/users/f0k/gists{/gist_id}","starred_url":"https://api.github.com/users/f0k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f0k/subscriptions","organizations_url":"https://api.github.com/users/f0k/orgs","repos_url":"https://api.github.com/users/f0k/repos","events_url":"https://api.github.com/users/f0k/events{/privacy}","received_events_url":"https://api.github.com/users/f0k/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-08-05T11:44:15Z","updated_at":"2015-08-06T10:18:09Z","closed_at":"2015-08-05T13:18:15Z","pull_request":{"url":"https://api.github.com/repos/Theano/Theano/pulls/3252","html_url":"https://github.com/Theano/Theano/pull/3252","diff_url":"https://github.com/Theano/Theano/pull/3252.diff","patch_url":"https://github.com/Theano/Theano/pull/3252.patch"},"body":"The convolution meta-optimizer currently fails for cuDNN with: `theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 0 is not part of the computational graph needed to compute the outputs: <CudaNdarrayType(float32, 4D)>.`\r\nThat's because it collects required inputs from the original GpuConv Op prior to applying any optimizers, and it may happen that the optimizers render one of the inputs unneeded (I haven't looked into this, but I assume it could be due to the output shape computation or something).\r\nThis PR fixes this by allowing the function compiled for timing the replaced Op to accept inputs it doesn't actually need (`on_unused_input='ignore'`)."},"comment":{"url":"https://api.github.com/repos/Theano/Theano/issues/comments/128316370","html_url":"https://github.com/Theano/Theano/pull/3252#issuecomment-128316370","issue_url":"https://api.github.com/repos/Theano/Theano/issues/3252","id":128316370,"user":{"login":"f0k","id":629706,"avatar_url":"https://avatars.githubusercontent.com/u/629706?v=3","gravatar_id":"","url":"https://api.github.com/users/f0k","html_url":"https://github.com/f0k","followers_url":"https://api.github.com/users/f0k/followers","following_url":"https://api.github.com/users/f0k/following{/other_user}","gists_url":"https://api.github.com/users/f0k/gists{/gist_id}","starred_url":"https://api.github.com/users/f0k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f0k/subscriptions","organizations_url":"https://api.github.com/users/f0k/orgs","repos_url":"https://api.github.com/users/f0k/repos","events_url":"https://api.github.com/users/f0k/events{/privacy}","received_events_url":"https://api.github.com/users/f0k/received_events","type":"User","site_admin":false},"created_at":"2015-08-06T10:18:09Z","updated_at":"2015-08-06T10:18:09Z","body":"> Which case did you found that conv_gemm was faster on the k40?\r\n\r\n```bash\r\n$ wget https://raw.githubusercontent.com/soumith/convnet-benchmarks/master/theano/pylearn2_benchmark.py\r\n$ SKIP=legacy,convnet,fft,meta python pylearn2_benchmark.py 1\r\nUsing gpu device 1: Tesla K40c (CNMeM is disabled)\r\n\r\nCONFIG: input = 3 x 128 x 128 * ker = 3 x 96 x 11 x 11 ( bs = 128 , stride = 1 )\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> fprop         ==>      78\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> bprop inputs  ==>     111\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> bprop weights ==>     118\r\n\r\n(auto) theano.sandbox.cuda.blas.GpuCorrMM          ==> fprop         ==>     119\r\n(auto) theano.sandbox.cuda.blas.GpuCorrMM          ==> bprop inputs  ==>     102\r\n(auto) theano.sandbox.cuda.blas.GpuCorrMM          ==> bprop weights ==>     242\r\n\r\n(manual) theano.sandbox.cuda.blas.GpuCorrMM        ==> fprop         ==>     119\r\n(manual) theano.sandbox.cuda.blas.GpuCorrMM        ==> bprop inputs  ==>     102\r\n(manual) theano.sandbox.cuda.blas.GpuCorrMM        ==> bprop weights ==>     241\r\n\r\n(manual conv) theano.sandbox.cuda.dnn.GpuDnnConv   ==> fprop         ==>      78\r\n(manual conv) theano.sandbox.cuda.dnn.GpuDnnConv   ==> bprop inputs  ==>     111\r\n(manual conv) theano.sandbox.cuda.dnn.GpuDnnConv   ==> bprop weights ==>     117\r\n```\r\n\r\nIt's the gradient wrt. inputs for a convolution of 3-channel images of 128x128 with 96 filters of 11x11, batchsize 128. That's the first layer in soumith's original convnet benchmarks. It's not a lot faster, but still.\r\n\r\n> I think nvidia would like to know it.\r\n\r\nDo they read here or do we need to send an email?\r\n\r\nAnother thing going wrong is the following:\r\n\r\n```bash\r\n$ THEANO_FLAGS=dnn.conv.algo_fwd=guess_on_shape_change SKIP=legacy,convnet,fft,meta,gemm python pylearn2_benchmark.py 1\r\nUsing gpu device 1: Tesla K40c (CNMeM is disabled)\r\n\r\nCONFIG: input = 3 x 128 x 128 * ker = 3 x 96 x 11 x 11 ( bs = 128 , stride = 1 )\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> fprop         ==>      93\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> bprop inputs  ==>     110\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> bprop weights ==>     118\r\n```\r\n\r\nThe heuristic gives a slower algorithm than the default one, and it's the FFT algorithm:\r\n\r\n```bash\r\n$ THEANO_FLAGS=dnn.conv.algo_fwd=fft SKIP=legacy,convnet,fft,meta,gemm python pylearn2_benchmark.py 1\r\nUsing gpu device 1: Tesla K40c (CNMeM is disabled)\r\n\r\nCONFIG: input = 3 x 128 x 128 * ker = 3 x 96 x 11 x 11 ( bs = 128 , stride = 1 )\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> fprop         ==>      94\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> bprop inputs  ==>     110\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> bprop weights ==>     117\r\n```\r\n\r\nSame performance as with guessing. Okay, the heuristics could be wrong / too simple, but timing should always be correct, shouldn't it? This needs editing of the `pylearn2_benchmark.py` file, from:\r\n```python\r\nnumber = 10  # nb of steps in loop to average over\r\nrepeat = 1   # nb of trials to pick the minimum of\r\n```\r\nto\r\n```python\r\nnumber = 3   # nb of steps in loop to average over\r\nrepeat = 3   # nb of trials to pick the minimum of\r\n```\r\nOtherwise, the first (costly) call will be included in the average. Let's see:\r\n```bash\r\n$ THEANO_FLAGS=dnn.conv.algo_fwd=time_on_shape_change SKIP=legacy,convnet,fft,meta,gemm python pylearn2_benchmark.py 1\r\nUsing gpu device 1: Tesla K40c (CNMeM is disabled)\r\n\r\nCONFIG: input = 3 x 128 x 128 * ker = 3 x 96 x 11 x 11 ( bs = 128 , stride = 1 )\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> fprop         ==>      95\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> bprop inputs  ==>     111\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> bprop weights ==>     118\r\n```\r\nAgain, it chooses the FFT algorithm, although that's not the fastest.\r\n\r\n-----\r\n\r\nFinally, and I'm separating this because it may be an issue with Theano:\r\n```bash\r\n$ THEANO_FLAGS=dnn.conv.algo_fwd=time_once,dnn.conv.algo_bwd=time_once SKIP=legacy,convnet,fft,meta,gemm python pylearn2_benchmark.py 1\r\nUsing gpu device 1: Tesla K40c (CNMeM is disabled)\r\n\r\nCONFIG: input = 3 x 128 x 128 * ker = 3 x 96 x 11 x 11 ( bs = 128 , stride = 1 )\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> fprop         ==>     512\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> bprop inputs  ==>    1879\r\n(auto) theano.sandbox.cuda.dnn.GpuDnnConv          ==> bprop weights ==>    1014\r\n\r\n(manual conv) theano.sandbox.cuda.dnn.GpuDnnConv   ==> fprop         ==>     512\r\n(manual conv) theano.sandbox.cuda.dnn.GpuDnnConv   ==> bprop inputs  ==>    1878\r\n(manual conv) theano.sandbox.cuda.dnn.GpuDnnConv   ==> bprop weights ==>    1013\r\n```\r\nThe `time_once` mode doesn't actually seem to only time once, but every time. Or it chooses the wrong algorithm. Or Theano doesn't call cuDNN correctly.\r\n\r\nSorry for hijacking this thread, but you asked here :) Shall I point NVIDIA to this, or are you in contact with them anyway? Or do you want to check whether it's a problem with Theano first?"}}