{"action":"created","issue":{"url":"https://api.github.com/repos/NVIDIA/caffe/issues/58","labels_url":"https://api.github.com/repos/NVIDIA/caffe/issues/58/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA/caffe/issues/58/comments","events_url":"https://api.github.com/repos/NVIDIA/caffe/issues/58/events","html_url":"https://github.com/NVIDIA/caffe/issues/58","id":114373793,"number":58,"title":"Benchmarking mode does not initialize CNMEM","user":{"login":"flx42","id":3645581,"avatar_url":"https://avatars.githubusercontent.com/u/3645581?v=3","gravatar_id":"","url":"https://api.github.com/users/flx42","html_url":"https://github.com/flx42","followers_url":"https://api.github.com/users/flx42/followers","following_url":"https://api.github.com/users/flx42/following{/other_user}","gists_url":"https://api.github.com/users/flx42/gists{/gist_id}","starred_url":"https://api.github.com/users/flx42/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/flx42/subscriptions","organizations_url":"https://api.github.com/users/flx42/orgs","repos_url":"https://api.github.com/users/flx42/repos","events_url":"https://api.github.com/users/flx42/events{/privacy}","received_events_url":"https://api.github.com/users/flx42/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-10-30T23:44:22Z","updated_at":"2015-10-31T00:30:23Z","closed_at":null,"body":"If you use branch caffe-0.14 and enable cuDNN, CNMEM will be enabled automatically. If you want to train a model using \"caffe.bin train\" then CNMEM will be initialized and allocation calls should virtually have zero cost. CNMEM is initialized here:\r\nhttps://github.com/NVIDIA/caffe/blob/caffe-0.14/tools/caffe.cpp#L200\r\n\r\nUnfortunately, if you want to benchmark this version of caffe using \"caffe.bin time\", CNMEM will not be initialized, it should be done here:\r\nhttps://github.com/NVIDIA/caffe/blob/caffe-0.14/tools/caffe.cpp#L316-L317\r\nSince the memory pool is not initialized, any allocation call will fallback to cudaMalloc:\r\nhttps://github.com/NVIDIA/caffe/blob/caffe-0.14/src/caffe/util/gpu_memory.cpp#L65-L66\r\n\r\nThe problem is that the new convolution calls attempts to do many allocations/deallocations:\r\nhttps://github.com/NVIDIA/caffe/blob/caffe-0.14/src/caffe/layers/cudnn_conv_layer.cu#L34\r\nIf we hit the cudaMalloc fallback, we do 1 cudaMalloc and 1 cudaFree **for each group and each convolution layer, for each forward pass.**\r\n\r\nThe original caffe code will be faster using \"caffe.bin time\" because it allocates the workspace only once:\r\nhttps://github.com/BVLC/caffe/blob/master/src/caffe/layers/cudnn_conv_layer.cpp#L200\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/NVIDIA/caffe/issues/comments/152679762","html_url":"https://github.com/NVIDIA/caffe/issues/58#issuecomment-152679762","issue_url":"https://api.github.com/repos/NVIDIA/caffe/issues/58","id":152679762,"user":{"login":"thatguymike","id":3503919,"avatar_url":"https://avatars.githubusercontent.com/u/3503919?v=3","gravatar_id":"","url":"https://api.github.com/users/thatguymike","html_url":"https://github.com/thatguymike","followers_url":"https://api.github.com/users/thatguymike/followers","following_url":"https://api.github.com/users/thatguymike/following{/other_user}","gists_url":"https://api.github.com/users/thatguymike/gists{/gist_id}","starred_url":"https://api.github.com/users/thatguymike/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatguymike/subscriptions","organizations_url":"https://api.github.com/users/thatguymike/orgs","repos_url":"https://api.github.com/users/thatguymike/repos","events_url":"https://api.github.com/users/thatguymike/events{/privacy}","received_events_url":"https://api.github.com/users/thatguymike/received_events","type":"User","site_admin":false},"created_at":"2015-10-31T00:30:23Z","updated_at":"2015-10-31T00:30:23Z","body":"Thanks Felix.  Actually, this bug is a little more complex than this.  The initial allocation should ONLY occur in GPU mode.  I think that explains another bug we have had for awhile.  Darn rebase/cross merge...\r\n"}}