{"action":"created","issue":{"url":"https://api.github.com/repos/Theano/Theano/issues/3084","labels_url":"https://api.github.com/repos/Theano/Theano/issues/3084/labels{/name}","comments_url":"https://api.github.com/repos/Theano/Theano/issues/3084/comments","events_url":"https://api.github.com/repos/Theano/Theano/issues/3084/events","html_url":"https://github.com/Theano/Theano/issues/3084","id":91720717,"number":3084,"title":"gpu illegal memory access","user":{"login":"zcyang","id":5890073,"avatar_url":"https://avatars.githubusercontent.com/u/5890073?v=3","gravatar_id":"","url":"https://api.github.com/users/zcyang","html_url":"https://github.com/zcyang","followers_url":"https://api.github.com/users/zcyang/followers","following_url":"https://api.github.com/users/zcyang/following{/other_user}","gists_url":"https://api.github.com/users/zcyang/gists{/gist_id}","starred_url":"https://api.github.com/users/zcyang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zcyang/subscriptions","organizations_url":"https://api.github.com/users/zcyang/orgs","repos_url":"https://api.github.com/users/zcyang/repos","events_url":"https://api.github.com/users/zcyang/events{/privacy}","received_events_url":"https://api.github.com/users/zcyang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-06-29T07:16:12Z","updated_at":"2015-06-29T18:46:08Z","closed_at":null,"body":"Hi,\r\n\r\nI meet with this problem when I am using theano for a project \r\n```shell\r\n2015-06-29T02:52:42.784295] iteration 0/368650 cost 285.914001 length 14\r\n2015-06-29T02:56:07.923116] validation cost: 298.046211\r\n13\r\n(10, 4096)\r\n2015-06-29T02:56:09.599871] iteration 1/368650 cost 300.855438 length 13\r\n12\r\n(10, 4096)\r\nError when tring to find the memory information on the GPU: an illegal memory access was encountered\r\nError freeing device pointer 0xb047d3800 (an illegal memory access was encountered). Driver report 0 bytes free and 0 bytes total\r\nError when tring to find the memory information on the GPU: an illegal memory access was encountered\r\nError freeing device pointer 0xb15e1e000 (an illegal memory access was encountered). Driver report 0 bytes free and 0 bytes total\r\ndevice_free: cudaFree() returned an error, but there is already an Python error set. This happen during the clean up when there is a first error and the CUDA driver is in a so bad state that it don't work anymore. We keep the previous error set to help debugging it.CudaNdarray_uninit: error freeing self->devdata. (self=0x20e35230, self->devata=0xb15e1e000)\r\nError when trying to find the memory information on the GPU: an illegal memory access was encountered\r\nError allocating 16777216 bytes of device memory (an illegal memory access was encountered). Driver report 0 bytes free and 0 bytes total\r\nTraceback (most recent call last):\r\n  File \"image_qa.py\", line 141, in <module>\r\n    label_idx, label_mask, label)\r\n  File \"/home/zichaoy/local/lib/python2.6/site-packages/Theano-0.7.0-py2.6.egg/theano/compile/function_module.py\", line 608, in __call__\r\n    storage_map=self.fn.storage_map)\r\n  File \"/home/zichaoy/local/lib/python2.6/site-packages/Theano-0.7.0-py2.6.egg/theano/compile/function_module.py\", line 597, in __call__\r\n    outputs = self.fn()\r\nMemoryError: Error allocating 16777216 bytes of device memory (an illegal memory access was encountered).\r\nApply node that caused the error: GpuElemwise{mul,no_inplace}(CudaNdarrayConstant{error while transferring the value: error copying data to host}, GpuDimShuffle{x,x}.0, image_hidden_w)\r\nInputs types: [CudaNdarrayType(float32, (True, True)), CudaNdarrayType(float32, (True, True)), CudaNdarrayType(float32, matrix)]\r\nInputs shapes: [(1, 1), (1, 1), (4096, 1024)]\r\nInputs strides: [(0, 0), (0, 0), (1024, 1)]\r\nInputs values: [<CudaNdarray object at 0x17a8c2f0>, <CudaNdarray object at 0x20e40e70>, 'not shown']\r\n```\r\n\r\nI am using k40 and I am sure that the memory is enough. I tried it on other types of gpus as well, the same problem occurs and it is deterministic. I have been using theano for several other projects and everything works out fine. \r\n\r\nI found other reports similar problems [here](https://groups.google.com/forum/#!msg/theano-users/Pu4YKlZKwm4/E7RdNHioXyoJ), however it seems no solution is provided. I have tried out all suggestion there as well, the error remains.\r\n\r\nCan it be a potential bug of theano?"},"comment":{"url":"https://api.github.com/repos/Theano/Theano/issues/comments/116793895","html_url":"https://github.com/Theano/Theano/issues/3084#issuecomment-116793895","issue_url":"https://api.github.com/repos/Theano/Theano/issues/3084","id":116793895,"user":{"login":"zcyang","id":5890073,"avatar_url":"https://avatars.githubusercontent.com/u/5890073?v=3","gravatar_id":"","url":"https://api.github.com/users/zcyang","html_url":"https://github.com/zcyang","followers_url":"https://api.github.com/users/zcyang/followers","following_url":"https://api.github.com/users/zcyang/following{/other_user}","gists_url":"https://api.github.com/users/zcyang/gists{/gist_id}","starred_url":"https://api.github.com/users/zcyang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zcyang/subscriptions","organizations_url":"https://api.github.com/users/zcyang/orgs","repos_url":"https://api.github.com/users/zcyang/repos","events_url":"https://api.github.com/users/zcyang/events{/privacy}","received_events_url":"https://api.github.com/users/zcyang/received_events","type":"User","site_admin":false},"created_at":"2015-06-29T18:46:08Z","updated_at":"2015-06-29T18:46:08Z","body":"It seems I fixed the problem.\r\n\r\nI change the matrix indexing from -1 to len(matrix) -1 and the problem disappears, but theano support negative indexing... not sure why..."}}