{"action":"created","issue":{"url":"https://api.github.com/repos/torch/cunn/issues/84","labels_url":"https://api.github.com/repos/torch/cunn/issues/84/labels{/name}","comments_url":"https://api.github.com/repos/torch/cunn/issues/84/comments","events_url":"https://api.github.com/repos/torch/cunn/issues/84/events","html_url":"https://github.com/torch/cunn/issues/84","id":72497027,"number":84,"title":"Nondeterministic behaviour in SpatialMaxPooling","user":{"login":"mys007","id":5921083,"avatar_url":"https://avatars.githubusercontent.com/u/5921083?v=3","gravatar_id":"","url":"https://api.github.com/users/mys007","html_url":"https://github.com/mys007","followers_url":"https://api.github.com/users/mys007/followers","following_url":"https://api.github.com/users/mys007/following{/other_user}","gists_url":"https://api.github.com/users/mys007/gists{/gist_id}","starred_url":"https://api.github.com/users/mys007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mys007/subscriptions","organizations_url":"https://api.github.com/users/mys007/orgs","repos_url":"https://api.github.com/users/mys007/repos","events_url":"https://api.github.com/users/mys007/events{/privacy}","received_events_url":"https://api.github.com/users/mys007/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-05-01T18:56:35Z","updated_at":"2015-05-01T19:48:40Z","closed_at":"2015-05-01T19:48:40Z","body":"I have encountered nondeterministic behavior in the backpropagation step of `SpatialMaxPooling` when `kW~=dW or kH~=dH` (i.e. `atomicmaxgradinput` kernel is run). I suspect the issue being caused by `atomicAdd()` and general non-associativity of floats. Thus, I'm fairly sure that this is a feature of parallelism but somehow I wanted to make you aware of it (it's scary)...\r\n\r\nThe following code works on my machine (GTX Titan, CUDA 346.46) when the GPU is under some load. If I let the computation run on CPU wit FloatTensors, it's deterministic. \r\n```\r\nlocal inp = torch.CudaTensor(1,18,18):zero()\r\ninp[1][3][3] = 1 --will force adding up 3 numbers\r\n       \r\nfw = torch.CudaTensor(1,8,8):fill(0)\r\nfw[1][2][1] = -0.00055786536540836\r\nfw[1][2][2] = 0.00075417151674628\r\nfw[1][1][2] = -0.00029314149287529\r\n\r\nlocal model = nn.SpatialMaxPooling(3, 3, 2, 2):cuda()    \r\nmodel:forward(inp)\r\nlocal bw = model:backward(inp, fw):clone()\r\n\r\nfor i=1,100 do\r\n\tlocal diff = bw - model:backward(inp, fw)\r\n\tprint(diff:sum()) --sometimes, the input is not zero!\r\nend\r\n```\r\n\r\nNote further that\r\n```\r\nlocal a = (-0.00055786536540836 + 0.00075417151674628) -0.00029314149287529\r\nlocal b = 0.00075417151674628 + (-0.00029314149287529 -0.00055786536540836)\r\nprint(a-b)  -- not zero\r\n```\r\n\r\nOn a slightly related note, is there any reason for calling `atomicmaxgradinput` instead of `maxgradinput` on SpatialMaxPooling.cu:320?"},"comment":{"url":"https://api.github.com/repos/torch/cunn/issues/comments/98218133","html_url":"https://github.com/torch/cunn/issues/84#issuecomment-98218133","issue_url":"https://api.github.com/repos/torch/cunn/issues/84","id":98218133,"user":{"login":"soumith","id":1310570,"avatar_url":"https://avatars.githubusercontent.com/u/1310570?v=3","gravatar_id":"","url":"https://api.github.com/users/soumith","html_url":"https://github.com/soumith","followers_url":"https://api.github.com/users/soumith/followers","following_url":"https://api.github.com/users/soumith/following{/other_user}","gists_url":"https://api.github.com/users/soumith/gists{/gist_id}","starred_url":"https://api.github.com/users/soumith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumith/subscriptions","organizations_url":"https://api.github.com/users/soumith/orgs","repos_url":"https://api.github.com/users/soumith/repos","events_url":"https://api.github.com/users/soumith/events{/privacy}","received_events_url":"https://api.github.com/users/soumith/received_events","type":"User","site_admin":false},"created_at":"2015-05-01T19:48:40Z","updated_at":"2015-05-01T19:48:40Z","body":"This is expected. Its Extremely hard to write a max pooling kernel that is deterministic when kW > dW or kH > dH. cunn, cudnn, caffe, all of these are non-deterministic maxpoolings for that case.\r\n\r\nwhen kW <= dW, the kernel can be made deterministic with some effort (I think it is deterministic, but I haven't verified).\r\nThe atomic adds were done at some point for just handling the case of kW != dW, for correctness, and they are needed. Otherwise two threads might race."}}