{"action":"created","issue":{"url":"https://api.github.com/repos/JuliaGPU/CUDArt.jl/issues/30","labels_url":"https://api.github.com/repos/JuliaGPU/CUDArt.jl/issues/30/labels{/name}","comments_url":"https://api.github.com/repos/JuliaGPU/CUDArt.jl/issues/30/comments","events_url":"https://api.github.com/repos/JuliaGPU/CUDArt.jl/issues/30/events","html_url":"https://github.com/JuliaGPU/CUDArt.jl/issues/30","id":107914324,"number":30,"title":"[Question] GPU->CPU Copy Speed?","user":{"login":"eric-tramel","id":1223539,"avatar_url":"https://avatars.githubusercontent.com/u/1223539?v=3","gravatar_id":"","url":"https://api.github.com/users/eric-tramel","html_url":"https://github.com/eric-tramel","followers_url":"https://api.github.com/users/eric-tramel/followers","following_url":"https://api.github.com/users/eric-tramel/following{/other_user}","gists_url":"https://api.github.com/users/eric-tramel/gists{/gist_id}","starred_url":"https://api.github.com/users/eric-tramel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eric-tramel/subscriptions","organizations_url":"https://api.github.com/users/eric-tramel/orgs","repos_url":"https://api.github.com/users/eric-tramel/repos","events_url":"https://api.github.com/users/eric-tramel/events{/privacy}","received_events_url":"https://api.github.com/users/eric-tramel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-09-23T13:04:03Z","updated_at":"2015-09-24T19:29:38Z","closed_at":null,"body":"I had a question about the use of `CUDArt.jl` in conjunction with `CuBLAS.jl`. I'm hoping to find out what I can do to improve the speed of memory copying from the GPU device back to CPU-accessible memory. I'm not sure if there is something I'm not implementing properly or if there is something intrinsic that I'm not properly understanding. I've conducted a series of tests using the BLAS Level-3 function, `gemm` which I detail below (exporting my IJulia notebook over to Markdown...) I should note that I'm using Julia v0.4-rc2 for these experiments.\r\n\r\nSpecifically, I've observed that for the implementation I detail below, the speed of the GPU->CPU `copy!` is about _2 orders of magnitude_ slower than the CPU->GPU copy. Surely I've done something wrong, right? If anyone could illuminate me on what I'm doing wrong, I'd be elated. \r\n\r\nThanks!\r\n\r\n\r\n# Testing CuBLAS and CUDArt for Julia\r\nAfter finally getting NVCC to work on OSX, we can start using the CUDA-themed BLAS packages written for Julia. In this notebook we will document how to utilize the necessary datatypes and show comparisons between the CPU and GPU implementations of common BLAS functions.\r\n\r\n## I. Calling and using the Libraries\r\nLets first make sure that we have updated and built the libraries. Because of the recent changes in Julia between `v0.3` and `v0.4`, we expect quite a number of warnings, and even errors, to pop up during the testing phase. However, the core functionality of the packges should be there.\r\n\r\n\r\n```julia\r\n# Update and build\r\nPkg.update()\r\nPkg.build(\"CUDArt\")\r\nPkg.build(\"CUBLAS\")\r\n```\r\n\r\n```julia\r\nusing CUDArt\r\nusing CUBLAS\r\nusing Base.LinAlg.BLAS\r\n```\r\n\r\n\r\n## II. Experiment Parameters\r\nWe will focus our comparisons on the BLAS function `gemm` which computes\r\n$$ \\mathbf{C} \\leftarrow \\alpha \\mathbf{A}\\mathbf{B} + \\beta \\mathbf{C}.$$\r\nWe will assume that all of these matrices are dense and real. For our experiments we will set\r\n$\\mathbf{A}: (n \\times m)$, $\\mathbf{B}: (m \\times k)$, $\\mathbf{C}: (n \\times k)$, and\r\n$\\alpha = \\beta = 1.0$.\r\n\r\n\r\n```julia\r\n# Dimensions\r\nn = 256\r\nm = 784\r\nk = 100\r\n# Scalings\r\na = 1.0\r\nb = 1.0\r\n# Initialization\r\nA = randn(n,m);\r\nB = randn(m,k);\r\nC = randn(n,k);\r\nwhos()\r\n```\r\n\r\n                                 A   1568 KB     256x784 Array{Float64,2} : [1.7596…\r\n                                 B    612 KB     784x100 Array{Float64,2} : [-1.596…\r\n                              Base  26665 KB     Module : Base\r\n                                 C    200 KB     256x100 Array{Float64,2} : [-0.344…\r\n                            CUBLAS    545 KB     Module : CUBLAS\r\n                            CUDArt    573 KB     Module : CUDArt\r\n                            Compat     58 KB     Module : Compat\r\n                              Core   3218 KB     Module : Core\r\n                    DataStructures    337 KB     Module : DataStructures\r\n                            IJulia    368 KB     Module : IJulia\r\n                    IPythonDisplay     26 KB     Module : IPythonDisplay\r\n                              JSON    195 KB     Module : JSON\r\n                              Main  33724 KB     Module : Main\r\n                           MyGemm!    966 bytes  Function : MyGemm!\r\n                      MyTimedGemm!     15 KB     Function : MyTimedGemm!\r\n                            Nettle    187 KB     Module : Nettle\r\n                               ZMQ     80 KB     Module : ZMQ\r\n                                 a      8 bytes  Float64 : 1.0\r\n                                 b      8 bytes  Float64 : 1.0\r\n                               d_A     40 bytes  CUDArt.CudaArray{Float64,2}(CUDArt…\r\n                               d_B     40 bytes  CUDArt.CudaArray{Float64,2}(CUDArt…\r\n                               d_C     40 bytes  CUDArt.CudaArray{Float64,2}(CUDArt…\r\n                               dev      8 bytes  Int64 : 0\r\n                                 k      8 bytes  Int64 : 100\r\n                                 m      8 bytes  Int64 : 784\r\n                                 n      8 bytes  Int64 : 256\r\n                            result      0 bytes  Void : nothing\r\n\r\n\r\n## III. Baseline Performance\r\nWe will now look at the timing of the base OpenBLAS implementation of `gemm`, which runs on the CPU, alone.\r\n\r\n\r\n```julia\r\n# Warmpup\r\ngemm!('N','N',a,A,B,b,C);\r\ngemm!('N','N',a,A,B,b,C);\r\n# Time: 5 runs\r\n@time gemm!('N','N',a,A,B,b,C);\r\n@time gemm!('N','N',a,A,B,b,C);\r\n@time gemm!('N','N',a,A,B,b,C);\r\n@time gemm!('N','N',a,A,B,b,C);\r\n@time gemm!('N','N',a,A,B,b,C);\r\n```\r\n\r\n      0.000769 seconds (4 allocations: 160 bytes)\r\n      0.000797 seconds (4 allocations: 160 bytes)\r\n      0.000810 seconds (4 allocations: 160 bytes)\r\n      0.000917 seconds (4 allocations: 160 bytes)\r\n      0.001528 seconds (4 allocations: 160 bytes)\r\n\r\n\r\n## IV. CUDArt Datatypes\r\nOur first step in being able to use CuBLAS is to initialize our GPU device and make on-device copies of the datastructures we're interested in. Below we detail how to fence off the GPU code and ensure that proper garbage collection is performed on the device via CUDArt.\r\n\r\n\r\n```julia\r\n# Assign Device\r\ndevice(0)\r\ndevice_reset(0) \r\ndevice(0)\r\n# Create and Copy \"A\"\r\nd_A = CudaArray(A)\r\ncopy!(d_A,A)\r\n# Create and Copy \"B\"\r\nd_B = CudaArray(B)\r\ncopy!(d_B,B)\r\n# Create and Copy \"C\"\r\nd_C = CudaArray(C)\r\ncopy!(d_C,C)\r\n# Show \r\nprintln(\"CUDArt Data Pointer Descriptions\")\r\nprintln(d_A)\r\nprintln(d_B)\r\nprintln(d_C)\r\n```\r\n\r\n    CUDArt Data Pointer Descriptions\r\n    CUDArt.CudaArray{Float64,2}(CUDArt.CudaPtr{Float64}(Ptr{Float64} @0x0000000d00a80000),(256,784),0)\r\n    CUDArt.CudaArray{Float64,2}(CUDArt.CudaPtr{Float64}(Ptr{Float64} @0x0000000d00c20000),(784,100),0)\r\n    CUDArt.CudaArray{Float64,2}(CUDArt.CudaPtr{Float64}(Ptr{Float64} @0x0000000d00d20000),(256,100),0)\r\n\r\n\r\n## V. CuBLAS Timings\r\nNow, lets look at the time requirements for just running `gemm`. We note that this **does not** include the time of memory copying to and from device memory. For now, lets limit ourselves to the direct comparison of the BLAS function implementation, alone.\r\n\r\n\r\n```julia\r\n# Warmpup\r\nCUBLAS.gemm!('N','N',a,d_A,d_B,b,d_C);\r\nCUBLAS.gemm!('N','N',a,d_A,d_B,b,d_C);\r\n# Time: 5 runs\r\n@time CUBLAS.gemm!('N','N',a,d_A,d_B,b,d_C);\r\n@time CUBLAS.gemm!('N','N',a,d_A,d_B,b,d_C);\r\n@time CUBLAS.gemm!('N','N',a,d_A,d_B,b,d_C);\r\n@time CUBLAS.gemm!('N','N',a,d_A,d_B,b,d_C);\r\n@time CUBLAS.gemm!('N','N',a,d_A,d_B,b,d_C);\r\n```\r\n\r\n      0.000033 seconds (24 allocations: 1.016 KB)\r\n      0.000053 seconds (24 allocations: 1.016 KB)\r\n      0.000053 seconds (24 allocations: 1.016 KB)\r\n      0.000045 seconds (24 allocations: 1.016 KB)\r\n      0.000037 seconds (24 allocations: 1.016 KB)\r\n\r\n\r\nSo, we can see form the above that we are looking at an *order of magnitude* improvement in computation time, potentially.\r\n\r\n\r\n```julia\r\n# End Session\r\ndevice_reset(0)\r\n```\r\n\r\n## VI. CuBLAS Timings: With Memory Copying\r\nWe will now look at the situation where we want to declare a local function which will conduct all of the necessary device-to-device memory copying requried for the GPU implemenation. Our goal is to see exactly how much advantage we retain in a realistic comparison.\r\n\r\n\r\n```julia\r\nfunction MyTimedGemm!(tA,tB,a,A,d_A,B,d_B,b,C,d_C)\r\n    # Copy to device\r\n    @printf \"(A->d_A)       \" \r\n        @time copy!(d_A,A)\r\n    @printf \"(B->d_B)       \" \r\n        @time copy!(d_B,B)\r\n    @printf \"(C->d_C)       \" \r\n        @time copy!(d_C,C)\r\n    # Run device-level BLAS\r\n    @printf \"(CUBLAS.gemm!) \"\r\n        @time CUBLAS.gemm!(tA,tB,a,d_A,d_B,b,d_C)\r\n    # Gather result\r\n    @printf \"(d_C->C)       \"\r\n        @time copy!(C,d_C)\r\nend\r\n\r\ndevice(0)\r\ndevice_reset(0)\r\ndevice(0)\r\n\r\n# These pointers can be pre-allocated\r\nd_A = CudaArray(A)\r\nd_B = CudaArray(B)\r\nd_C = CudaArray(C)\r\n\r\n# Warmup\r\nprintln(\"Warmups============\")\r\nMyTimedGemm!('N','N',a,A,d_A,B,d_B,b,C,d_C);\r\nMyTimedGemm!('N','N',a,A,d_A,B,d_B,b,C,d_C);\r\nprintln(\"Actual=============\")\r\n@time MyTimedGemm!('N','N',a,A,d_A,B,d_B,b,C,d_C);\r\n```\r\n\r\n    Warmups============\r\n    (A->d_A)         0.000204 seconds\r\n    (B->d_B)         0.000317 seconds\r\n    (C->d_C)         0.000075 seconds\r\n    (CUBLAS.gemm!)   0.078434 seconds (20 allocations: 880 bytes)\r\n    (d_C->C)         0.006428 seconds\r\n    (A->d_A)         0.000442 seconds\r\n    (B->d_B)         0.000361 seconds\r\n    (C->d_C)         0.000063 seconds\r\n    (CUBLAS.gemm!)   0.000043 seconds (20 allocations: 880 bytes)\r\n    (d_C->C)         0.006849 seconds\r\n    Actual=============\r\n    (A->d_A)         0.000214 seconds\r\n    (B->d_B)         0.000307 seconds\r\n    (C->d_C)         0.000076 seconds\r\n    (CUBLAS.gemm!)   0.000038 seconds (20 allocations: 880 bytes)\r\n    (d_C->C)         0.007070 seconds\r\n      0.008016 seconds (199 allocations: 7.813 KB)\r\n\r\n\r\nWe can see that the act of reading the matrix $\\mathbf{C}$ back from the device to the CPU actually incurs a huge cost. In fact, the cost is so high as to entirely remove any time advantage we obtain from the CuBLAS implemenation of `gemm`. "},"comment":{"url":"https://api.github.com/repos/JuliaGPU/CUDArt.jl/issues/comments/143026302","html_url":"https://github.com/JuliaGPU/CUDArt.jl/issues/30#issuecomment-143026302","issue_url":"https://api.github.com/repos/JuliaGPU/CUDArt.jl/issues/30","id":143026302,"user":{"login":"lucasb-eyer","id":1476029,"avatar_url":"https://avatars.githubusercontent.com/u/1476029?v=3","gravatar_id":"","url":"https://api.github.com/users/lucasb-eyer","html_url":"https://github.com/lucasb-eyer","followers_url":"https://api.github.com/users/lucasb-eyer/followers","following_url":"https://api.github.com/users/lucasb-eyer/following{/other_user}","gists_url":"https://api.github.com/users/lucasb-eyer/gists{/gist_id}","starred_url":"https://api.github.com/users/lucasb-eyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lucasb-eyer/subscriptions","organizations_url":"https://api.github.com/users/lucasb-eyer/orgs","repos_url":"https://api.github.com/users/lucasb-eyer/repos","events_url":"https://api.github.com/users/lucasb-eyer/events{/privacy}","received_events_url":"https://api.github.com/users/lucasb-eyer/received_events","type":"User","site_admin":false},"created_at":"2015-09-24T19:29:38Z","updated_at":"2015-09-24T19:29:38Z","body":"You probably have to change your timing lines from `@time copy!(d_B,B)` into something like\r\n\r\n```\r\ndevice_synchronize() ; @time (copy!(d_B,B) ; device_synchronize())\r\n```\r\n\r\nPS: I would be grateful of a copy of the full notebook (maybe as a gist) if possible, it looks like a good starting point for experimentation!"}}