{"action":"created","issue":{"url":"https://api.github.com/repos/torch/cutorch/issues/42","labels_url":"https://api.github.com/repos/torch/cutorch/issues/42/labels{/name}","comments_url":"https://api.github.com/repos/torch/cutorch/issues/42/comments","events_url":"https://api.github.com/repos/torch/cutorch/issues/42/events","html_url":"https://github.com/torch/cutorch/issues/42","id":43657470,"number":42,"title":"Multi-GPU support","user":{"login":"soumith","id":1310570,"avatar_url":"https://avatars.githubusercontent.com/u/1310570?v=3","gravatar_id":"","url":"https://api.github.com/users/soumith","html_url":"https://github.com/soumith","followers_url":"https://api.github.com/users/soumith/followers","following_url":"https://api.github.com/users/soumith/following{/other_user}","gists_url":"https://api.github.com/users/soumith/gists{/gist_id}","starred_url":"https://api.github.com/users/soumith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumith/subscriptions","organizations_url":"https://api.github.com/users/soumith/orgs","repos_url":"https://api.github.com/users/soumith/repos","events_url":"https://api.github.com/users/soumith/events{/privacy}","received_events_url":"https://api.github.com/users/soumith/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":34,"created_at":"2014-09-23T17:18:13Z","updated_at":"2015-08-13T22:12:38Z","closed_at":"2014-09-25T15:42:29Z","body":"MultiGPU support has been implemented in cutorch (and by extension all torch cuda libraries like cunn, cudnn etc.).\r\n\r\n* Switch the device on the fly with cutorch.setDevice(devID)\r\n* All cuda calls are asynchronous, and can be synchronized with cutorch.synchronize()\r\n\r\nExample usage for tensors:\r\n``` lua\r\n-- Let us do matrix addition for matrices sitting on two different GPUs\r\ncutorch.setDevice(1)\r\nmatrix1 = torch.CudaTensor(10):fill(1)\r\nprint(matrix1) -- printing is a synchronous call, so you dont have to explicitly call cutorch.synchronize()\r\ncutorch.setDevice(2)\r\nmatrix2 = torch.CudaTensor(10):fill(2)\r\nprint(matrix2) \r\nmatrix2:add(matrix1) -- matrix1 is seamlessly copied onto GPU2 and added to matrix2\r\nprint(matrix2)\r\n``` \r\n\r\nif you want to do data-parallel training of neural nets (including convnets), your training loop can run like this:\r\n\r\nFor each mini-batch: \r\n``` lua\r\n1. load data (preferably using multiple threads, for example using [threads-ffi](https://github.com/torch/threads-ffi))\r\n2. loop over GPUs (the loop below will be completely anynchronous, so will run parallely)\r\n  2.1. model[gpuX]:forward\r\n  2.2. criterion[gpuX]:forward\r\n  2.3. criterion[gpuX]:backward\r\n  2.4. model[gpuX]:backward\r\n3. cutorch.synchronize()\r\n4. accumulate GPUx's gradParameters to GPU1's gradParameters\r\n5. do SGD on GPU1\r\n6. copy back GPU1's parameters to GPUx\r\n7. cutorch.synchronize() and print accuracy etc.\r\n```\r\nLoop back to 1 for next mini-batch\r\n\r\nAlso, to train ConvNets using multiple GPUs, I recommend using [CuDNN](https://github.com/soumith/cudnn.torch) for the convolution layers, as I've tested that they are completely asynchronous (meaning that the processing runs parallely on multiple GPUs)\r\n\r\n**Comments below describe the technical details of changes made. If you just want to use Multi-GPU, you can stop reading now.** "},"comment":{"url":"https://api.github.com/repos/torch/cutorch/issues/comments/130863618","html_url":"https://github.com/torch/cutorch/issues/42#issuecomment-130863618","issue_url":"https://api.github.com/repos/torch/cutorch/issues/42","id":130863618,"user":{"login":"soumith","id":1310570,"avatar_url":"https://avatars.githubusercontent.com/u/1310570?v=3","gravatar_id":"","url":"https://api.github.com/users/soumith","html_url":"https://github.com/soumith","followers_url":"https://api.github.com/users/soumith/followers","following_url":"https://api.github.com/users/soumith/following{/other_user}","gists_url":"https://api.github.com/users/soumith/gists{/gist_id}","starred_url":"https://api.github.com/users/soumith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumith/subscriptions","organizations_url":"https://api.github.com/users/soumith/orgs","repos_url":"https://api.github.com/users/soumith/repos","events_url":"https://api.github.com/users/soumith/events{/privacy}","received_events_url":"https://api.github.com/users/soumith/received_events","type":"User","site_admin":false},"created_at":"2015-08-13T22:12:38Z","updated_at":"2015-08-13T22:12:38Z","body":"@algred the only operation that is allowed cross-GPU is the copy operation. All other operations are checked with assertions to make sure what you are doing is not possible. To get good performance, you'll have to copy matrix1 onto matrix2's GPU and then do the mathematical operation.\r\n\r\nIf you dont like this setting, you can simply disable these assertions by adding the define DISABLE_CHECK_GPU and reinstalling cutorch.\r\n\r\nhttps://github.com/torch/cutorch/blob/master/lib/THC/THCTensor.c#L761"}}