{"action":"created","issue":{"url":"https://api.github.com/repos/Theano/Theano/issues/2607","labels_url":"https://api.github.com/repos/Theano/Theano/issues/2607/labels{/name}","comments_url":"https://api.github.com/repos/Theano/Theano/issues/2607/comments","events_url":"https://api.github.com/repos/Theano/Theano/issues/2607/events","html_url":"https://github.com/Theano/Theano/issues/2607","id":60874981,"number":2607,"title":"inplace dnn convolution seems dangerously broken","user":{"login":"f0k","id":629706,"avatar_url":"https://avatars.githubusercontent.com/u/629706?v=3","gravatar_id":"","url":"https://api.github.com/users/f0k","html_url":"https://github.com/f0k","followers_url":"https://api.github.com/users/f0k/followers","following_url":"https://api.github.com/users/f0k/following{/other_user}","gists_url":"https://api.github.com/users/f0k/gists{/gist_id}","starred_url":"https://api.github.com/users/f0k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f0k/subscriptions","organizations_url":"https://api.github.com/users/f0k/orgs","repos_url":"https://api.github.com/users/f0k/repos","events_url":"https://api.github.com/users/f0k/events{/privacy}","received_events_url":"https://api.github.com/users/f0k/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/Theano/Theano/milestones/3","html_url":"https://github.com/Theano/Theano/milestones/0.7","labels_url":"https://api.github.com/repos/Theano/Theano/milestones/3/labels","id":112020,"number":3,"title":"0.7","description":"","creator":{"login":"nouiz","id":180987,"avatar_url":"https://avatars.githubusercontent.com/u/180987?v=3","gravatar_id":"","url":"https://api.github.com/users/nouiz","html_url":"https://github.com/nouiz","followers_url":"https://api.github.com/users/nouiz/followers","following_url":"https://api.github.com/users/nouiz/following{/other_user}","gists_url":"https://api.github.com/users/nouiz/gists{/gist_id}","starred_url":"https://api.github.com/users/nouiz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nouiz/subscriptions","organizations_url":"https://api.github.com/users/nouiz/orgs","repos_url":"https://api.github.com/users/nouiz/repos","events_url":"https://api.github.com/users/nouiz/events{/privacy}","received_events_url":"https://api.github.com/users/nouiz/received_events","type":"User","site_admin":false},"open_issues":35,"closed_issues":54,"state":"open","created_at":"2012-04-25T19:38:54Z","updated_at":"2015-03-13T22:01:58Z","due_on":null,"closed_at":null},"comments":24,"created_at":"2015-03-12T18:49:28Z","updated_at":"2015-03-14T23:16:18Z","closed_at":null,"body":"I've recently noticed that all my experiments run slower (which I attributed to other processes running on our server) and that I couldn't reproduce results (which I attributed to said experiments being in an early stage with possibly badly chosen hyperparameters, such that earlier runs might just have been lucky). After a while, as I couldn't get any network to give reasonable predictions and ruled out anything else I've changed in between, I've looked into my `git reflog` for Theano and undid the last update. Io and behold, everything was running three times faster and networks learned as they ought to.\r\n\r\nBisecting, I found that the offending commit is a0964ac, the merge commit for #2559, which introduced inplace operations for cuDNN convolutions along with the necessary optimizers. Profiling that commit and the one immediately before, I get:\r\n```\r\nat commit b1963ef (before #2559):\r\n\r\n<% time> <sum %> <apply time> <time per call> <#call> <id> <Apply name>\r\n  12.3%    12.3%       1.670s       4.09e-03s    408   199   GpuDnnConvGradW(GpuContiguous.0, GpuContiguous.0, GpuDnnConvDesc{border_mode='valid', subsample=(1, 1), conv_mode='cross'}.0, ScalarFromTensor.0, ScalarFromTensor.0)\r\n   9.2%    21.4%       1.245s       3.05e-03s    408    61   GpuDnnConv{workmem='small'}(CopyOnNegativeStrides.0, GpuContiguous.0, GpuDnnConvDesc{border_mode='valid', subsample=(1, 1), conv_mode='conv'}.0)\r\n   6.6%    28.0%       0.897s       2.20e-03s    408   154   GpuDnnPoolGrad(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuDnnPoolDesc{ws=(3, 6), stride=(3, 6), mode='max', pad=(0, 0)}.0)\r\n   6.4%    34.4%       0.869s       2.13e-03s    408   147   GpuDnnConvGradI(GpuContiguous.0, GpuContiguous.0, GpuDnnConvDesc{border_mode='valid', subsample=(1, 1), conv_mode='cross'}.0, ScalarFromTensor.0, ScalarFromTensor.0)\r\n\r\n\r\nat commit a0964ac (after #2559):\r\n\r\n<% time> <sum %> <apply time> <time per call> <#call> <id> <Apply name>\r\n  26.2%    26.2%      11.450s       2.84e-02s    403    76   HostFromGpu(GpuElemwise{Add}[(0, 1)].0)\r\n  25.0%    51.3%      10.921s       2.71e-02s    403    78   GpuFromHost(HostFromGpu.0)\r\n   3.7%    54.9%       1.596s       3.96e-03s    403   201   GpuDnnConvGradW{inplace=True}(CopyOnNegativeStrides.0, GpuContiguous.0, GpuAlloc{memset_0=True}.0, GpuDnnConvDesc{border_mode='valid', subsample=(1, 1), conv_mode='cross'}.0, Constant{1.0})\r\n   3.6%    58.5%       1.554s       3.86e-03s    403    80   GpuDnnConv{workmem='small', inplace=True}(CopyOnNegativeStrides.0, GpuContiguous.0, GpuElemwise{Composite{(Abs(i0) + i1)}}[(0, 1)].0, GpuDnnConvDesc{border_mode='valid', subsample=(1, 1), conv_mode='conv'}.0, Constant{0.5})\r\n   3.0%    61.5%       1.321s       3.28e-03s    403    77   GpuDnnConv{workmem='small', inplace=True}(CopyOnNegativeStrides.0, GpuContiguous.0, GpuElemwise{Add}[(0, 1)].0, GpuDnnConvDesc{border_mode='valid', subsample=(1, 1), conv_mode='conv'}.0, Constant{1.0})\r\n```\r\n\r\nThat is, it is copying data from the device to the host just to copy it back to the device. Furthermore, the networks trained after that commit tend to output zeros only when computing validation or test data predictions, while training seems to progress normally (judging from the training error). This indicates that something is wrong with the computations; something not caught by the tests.\r\n\r\nFor lack of time I'll just continue working with b1963ef for now. I fear I cannot easily turn my code into a minimal working example, but I'll happily answer questions that aid in debugging this (enabling/disabling optimizations, printing the graph before/after optimization, etc)."},"comment":{"url":"https://api.github.com/repos/Theano/Theano/issues/comments/80758394","html_url":"https://github.com/Theano/Theano/issues/2607#issuecomment-80758394","issue_url":"https://api.github.com/repos/Theano/Theano/issues/2607","id":80758394,"user":{"login":"f0k","id":629706,"avatar_url":"https://avatars.githubusercontent.com/u/629706?v=3","gravatar_id":"","url":"https://api.github.com/users/f0k","html_url":"https://github.com/f0k","followers_url":"https://api.github.com/users/f0k/followers","following_url":"https://api.github.com/users/f0k/following{/other_user}","gists_url":"https://api.github.com/users/f0k/gists{/gist_id}","starred_url":"https://api.github.com/users/f0k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f0k/subscriptions","organizations_url":"https://api.github.com/users/f0k/orgs","repos_url":"https://api.github.com/users/f0k/repos","events_url":"https://api.github.com/users/f0k/events{/privacy}","received_events_url":"https://api.github.com/users/f0k/received_events","type":"User","site_admin":false},"created_at":"2015-03-14T23:16:18Z","updated_at":"2015-03-14T23:16:18Z","body":"And regarding the performance difference, are the optimizations executed late enough so it's sure there are no other clients in the graph that need the convolution results? Maybe some convolutions are computed twice now because they cannot be merged any more with a different alpha or an inplace version? (Just throwing in some ideas.)"}}