{"action":"created","issue":{"url":"https://api.github.com/repos/BVLC/caffe/issues/2195","labels_url":"https://api.github.com/repos/BVLC/caffe/issues/2195/labels{/name}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/2195/comments","events_url":"https://api.github.com/repos/BVLC/caffe/issues/2195/events","html_url":"https://github.com/BVLC/caffe/pull/2195","id":64240527,"number":2195,"title":"OpenCL Backend","user":{"login":"lunochod","id":2642598,"avatar_url":"https://avatars.githubusercontent.com/u/2642598?v=3","gravatar_id":"","url":"https://api.github.com/users/lunochod","html_url":"https://github.com/lunochod","followers_url":"https://api.github.com/users/lunochod/followers","following_url":"https://api.github.com/users/lunochod/following{/other_user}","gists_url":"https://api.github.com/users/lunochod/gists{/gist_id}","starred_url":"https://api.github.com/users/lunochod/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lunochod/subscriptions","organizations_url":"https://api.github.com/users/lunochod/orgs","repos_url":"https://api.github.com/users/lunochod/repos","events_url":"https://api.github.com/users/lunochod/events{/privacy}","received_events_url":"https://api.github.com/users/lunochod/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/BVLC/caffe/labels/compatibility","name":"compatibility","color":"fbe58a"},{"url":"https://api.github.com/repos/BVLC/caffe/labels/enhancement","name":"enhancement","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":28,"created_at":"2015-03-25T10:48:09Z","updated_at":"2015-04-12T19:54:59Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/BVLC/caffe/pulls/2195","html_url":"https://github.com/BVLC/caffe/pull/2195","diff_url":"https://github.com/BVLC/caffe/pull/2195.diff","patch_url":"https://github.com/BVLC/caffe/pull/2195.patch"},"body":"# About\r\n\r\nThe proposed changes add OpenCL support to Caffe. All GPU functions can be executed using AMD GPUs w/ OpenCL 1.2 or 2.0 as well as nVidia GPUs w/ OpenCL 1.1.\r\n\r\n# OpenCL Build Target\r\n\r\nEdit Makefile and Makefile.config.OpenCL to enable OpenCL as a build target. clBLAS is required to support BLAS functions through OpenCL.\r\n\r\n# OpenCL Tests\r\n\r\nAll GPU tests successfully complete using this OpenCL version of Caffe.\r\n\r\n# Performance and Stability\r\n\r\nThe main goal was to provide an OpenCL port to the Caffe community. As such it is not yet optimized for performance or stability.\r\n\r\n# Help Wanted\r\n\r\nLet's make it better and faster together.  "},"comment":{"url":"https://api.github.com/repos/BVLC/caffe/issues/comments/92109543","html_url":"https://github.com/BVLC/caffe/pull/2195#issuecomment-92109543","issue_url":"https://api.github.com/repos/BVLC/caffe/issues/2195","id":92109543,"user":{"login":"jyegerlehner","id":2138320,"avatar_url":"https://avatars.githubusercontent.com/u/2138320?v=3","gravatar_id":"","url":"https://api.github.com/users/jyegerlehner","html_url":"https://github.com/jyegerlehner","followers_url":"https://api.github.com/users/jyegerlehner/followers","following_url":"https://api.github.com/users/jyegerlehner/following{/other_user}","gists_url":"https://api.github.com/users/jyegerlehner/gists{/gist_id}","starred_url":"https://api.github.com/users/jyegerlehner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jyegerlehner/subscriptions","organizations_url":"https://api.github.com/users/jyegerlehner/orgs","repos_url":"https://api.github.com/users/jyegerlehner/repos","events_url":"https://api.github.com/users/jyegerlehner/events{/privacy}","received_events_url":"https://api.github.com/users/jyegerlehner/received_events","type":"User","site_admin":false},"created_at":"2015-04-12T19:54:59Z","updated_at":"2015-04-12T19:54:59Z","body":"Thanks bhack for the prod on AMD's optimization guidelines. \r\n\r\nMy opencl GPU benchmark on the MNIST example mentioned above shows we're spending almost all our time in the convolution layer:\r\n\r\n```\r\nI0411 22:24:44.741221  5701 caffe.cpp:314] Average time per layer: \r\nI0411 22:24:44.741237  5701 caffe.cpp:317]      mnist\tforward: 0.341 ms.\r\nI0411 22:24:44.741255  5701 caffe.cpp:320]      mnist\tbackward: 0.00096 ms.\r\nI0411 22:24:44.741271  5701 caffe.cpp:317]      conv1\tforward: 13.7345 ms.\r\nI0411 22:24:44.741287  5701 caffe.cpp:320]      conv1\tbackward: 27.9414 ms.\r\nI0411 22:24:44.741304  5701 caffe.cpp:317]      pool1\tforward: 0.04424 ms.\r\nI0411 22:24:44.741319  5701 caffe.cpp:320]      pool1\tbackward: 0.04526 ms.\r\nI0411 22:24:44.741336  5701 caffe.cpp:317]      conv2\tforward: 35.0272 ms.\r\nI0411 22:24:44.741353  5701 caffe.cpp:320]      conv2\tbackward: 37.0192 ms.\r\nI0411 22:24:44.741367  5701 caffe.cpp:317]      pool2\tforward: 0.0485 ms.\r\nI0411 22:24:44.741384  5701 caffe.cpp:320]      pool2\tbackward: 0.01406 ms.\r\nI0411 22:24:44.741402  5701 caffe.cpp:317]        ip1\tforward: 1.09904 ms.\r\nI0411 22:24:44.741418  5701 caffe.cpp:320]        ip1\tbackward: 0.04586 ms.\r\nI0411 22:24:44.741432  5701 caffe.cpp:317]      relu1\tforward: 0.02002 ms.\r\nI0411 22:24:44.741447  5701 caffe.cpp:320]      relu1\tbackward: 0.01212 ms.\r\nI0411 22:24:44.741462  5701 caffe.cpp:317]        ip2\tforward: 0.3685 ms.\r\nI0411 22:24:44.741477  5701 caffe.cpp:320]        ip2\tbackward: 0.0516 ms.\r\nI0411 22:24:44.741493  5701 caffe.cpp:317]       loss\tforward: 1.88218 ms.\r\nI0411 22:24:44.741514  5701 caffe.cpp:320]       loss\tbackward: 0.38836 ms.\r\nI0411 22:24:44.741530  5701 caffe.cpp:325] Average Forward pass: 52.5792 ms.\r\nI0411 22:24:44.741540  5701 caffe.cpp:327] Average Backward pass: 65.5275 ms.\r\nI0411 22:24:44.741551  5701 caffe.cpp:329] Average Forward-Backward: 118.3 ms.\r\nI0411 22:24:44.741562  5701 caffe.cpp:331] Total Time: 5915 ms.\r\nI0411 22:24:44.741575  5701 caffe.cpp:332] *** Benchmark ends ***\r\n```\r\n\r\nThe convolution layer's forward_gpu has this loop in it:\r\n```c++\r\n    for (int n = 0; n < this->num_; ++n) {\r\n      TIME(\"forward_gpu_gemm()\", {\r\n          this->forward_gpu_gemm(bottom_data + bottom[i]->offset(n), weight, top_data + top[i]->offset(n));\r\n      });\r\n      if (this->bias_term_) {\r\n        const Dtype* bias = this->blobs_[1]->gpu_data();\r\n        TIME(\"forward_gpu_bias()\", {\r\n            this->forward_gpu_bias(top_data + top[i]->offset(n), bias);\r\n        });\r\n      }\r\n    }\r\n```\r\nwhere loop variable n is looping over the number of items in the minibatch, which is 100 in the mnist example. So each time through the loop we are enqueuing a GPU im2col and a GEMM operation into a single in-order executed command queue. Each of the 100 is executed one at a time until they're all done.  As lunochod pointed out, opencl doesn't perform very well on smaller GEMM ops as presumably the GPU isn't saturated.  The optimization guide suggests enqueueing smaller kernels across multiple command queues. The R9-290X has 8 hardware command queues so we could get presumably up-to an 8X speed up that way. Or possibly the bottleneck is the overhead of setting the kernel arguments and enqueueing for each of the 100 kernel launches, in which case we wouldn't see so much improvement.\r\n\r\nI tried hacking in a pool of command queues (instead of a single command queue) to see if we get any speedup. I pass the loop variable n down to the point where we enqueue the im2col and GEMM operations and use it to rotate through the command queues. Thought I had the queues synchronized with clFinish calls, but haven't got all the tests passing yet. \r\n\r\nHugoFeng's kernel is interesting. I don't understand the whole thing yet, but I like the idea that it reduces required memory bandwidth by loading the convolution filter weights from global memory only once (storing them in  private memory), then using them for all the calculations to be done. It doesn't have the concept of stride, groups or rectangular filters from what I see. If we are going to have a custom kernel for the OpenCL version of convolution, I would think we'd want a single kernel launch that does the whole minibatch. \r\n\r\n"}}