{"action":"created","issue":{"url":"https://api.github.com/repos/BVLC/caffe/issues/2768","labels_url":"https://api.github.com/repos/BVLC/caffe/issues/2768/labels{/name}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/2768/comments","events_url":"https://api.github.com/repos/BVLC/caffe/issues/2768/events","html_url":"https://github.com/BVLC/caffe/issues/2768","id":95093269,"number":2768,"title":"cudaMemcpy take 16.4% caffe cpu usage,  is that too high? ","user":{"login":"smartbitcoin","id":5984485,"avatar_url":"https://avatars.githubusercontent.com/u/5984485?v=3","gravatar_id":"","url":"https://api.github.com/users/smartbitcoin","html_url":"https://github.com/smartbitcoin","followers_url":"https://api.github.com/users/smartbitcoin/followers","following_url":"https://api.github.com/users/smartbitcoin/following{/other_user}","gists_url":"https://api.github.com/users/smartbitcoin/gists{/gist_id}","starred_url":"https://api.github.com/users/smartbitcoin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smartbitcoin/subscriptions","organizations_url":"https://api.github.com/users/smartbitcoin/orgs","repos_url":"https://api.github.com/users/smartbitcoin/repos","events_url":"https://api.github.com/users/smartbitcoin/events{/privacy}","received_events_url":"https://api.github.com/users/smartbitcoin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-07-15T03:02:22Z","updated_at":"2015-07-21T20:24:26Z","closed_at":null,"body":"![screenshot from 2015-07-14 22 56 02](https://cloud.githubusercontent.com/assets/5984485/8689905/c1d6e19e-2a7b-11e5-83e2-8e8e70a15a59.png)\r\n\r\nI trained a net model most of full connected layer.  when I do profile for training on GPU ,  I saw that cudaMemcpy take near 16% CPU usage.\r\nthere should be plenty of space for optimize this.\r\nOne thoughts in my mind is that the layer if it's for GPU,   it should located in __device__ function and there should be a gpu resolver which as a __global__ cuda function for resolve it  to save the cost cudaMemcpy. \r\n\r\nare there anybody have profile experience can shared in here? \r\n"},"comment":{"url":"https://api.github.com/repos/BVLC/caffe/issues/comments/123467896","html_url":"https://github.com/BVLC/caffe/issues/2768#issuecomment-123467896","issue_url":"https://api.github.com/repos/BVLC/caffe/issues/2768","id":123467896,"user":{"login":"wafflette","id":11221446,"avatar_url":"https://avatars.githubusercontent.com/u/11221446?v=3","gravatar_id":"","url":"https://api.github.com/users/wafflette","html_url":"https://github.com/wafflette","followers_url":"https://api.github.com/users/wafflette/followers","following_url":"https://api.github.com/users/wafflette/following{/other_user}","gists_url":"https://api.github.com/users/wafflette/gists{/gist_id}","starred_url":"https://api.github.com/users/wafflette/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wafflette/subscriptions","organizations_url":"https://api.github.com/users/wafflette/orgs","repos_url":"https://api.github.com/users/wafflette/repos","events_url":"https://api.github.com/users/wafflette/events{/privacy}","received_events_url":"https://api.github.com/users/wafflette/received_events","type":"User","site_admin":false},"created_at":"2015-07-21T20:24:26Z","updated_at":"2015-07-21T20:24:26Z","body":"Does your entire model fit into GPU memory?\r\n\r\nIt seems if it fits and your whole model stays in GPU memory all the time then theoretically I agree, there should be almost no cost associated with cudaMemcpy.\r\n\r\n\r\n"}}