{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/675","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/675/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/675/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/675/events","html_url":"https://github.com/dmlc/mxnet/issues/675","id":118221468,"number":675,"title":"GPU memory issue","user":{"login":"Strateus","id":5196895,"avatar_url":"https://avatars.githubusercontent.com/u/5196895?v=3","gravatar_id":"","url":"https://api.github.com/users/Strateus","html_url":"https://github.com/Strateus","followers_url":"https://api.github.com/users/Strateus/followers","following_url":"https://api.github.com/users/Strateus/following{/other_user}","gists_url":"https://api.github.com/users/Strateus/gists{/gist_id}","starred_url":"https://api.github.com/users/Strateus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Strateus/subscriptions","organizations_url":"https://api.github.com/users/Strateus/orgs","repos_url":"https://api.github.com/users/Strateus/repos","events_url":"https://api.github.com/users/Strateus/events{/privacy}","received_events_url":"https://api.github.com/users/Strateus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-11-21T21:18:15Z","updated_at":"2015-11-22T03:13:29Z","closed_at":null,"body":"Im not sure if its a bug or not, i loaded a model from caffe zoo (pre-converted to mxnet format):\r\n```\r\n>>> import mxnet as mx\r\n>>> import time\r\n>>> model = mx.model.FeedForward.load('/root/models/mxnet_vgg_face', 1, numpy_batch_size=2, ctx=[mx.gpu(0)]) \r\n```\r\nmade iterator of 11 images, batch size = 2\r\n```\r\ndataiter = mx.io.ImageRecordIter(path_imgrec=\"/root/test_images/output.bin\", data_shape=(3,224,224), path_imglist=\"/root/test_images/list.txt\", batch_size = 2)\r\n```\r\nbefore start checked my GPU memory, im not sure whats MiB but its almost empty and my card should have 1Gb:\r\n```\r\nroot@gpu:~/mxnet# nvidia-smi\r\nSun Nov 22 00:03:16 2015       \r\n+------------------------------------------------------+                       \r\n| NVIDIA-SMI 352.39     Driver Version: 352.39         |                       \r\n|-------------------------------+----------------------+----------------------+\r\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n|===============================+======================+======================|\r\n|   0  GeForce GTX 650     Off  | 0000:01:00.0     N/A |                  N/A |\r\n| 40%   36C    P8    N/A /  N/A |     53MiB /  2046MiB |     N/A      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n```\r\nThen CUDA said im out of memory:\r\n```\r\n>>> t1=time.time(); res = model.predict(dataiter); print 'Timing (per image): %sms' % str(((time.time()-t1)*1000))\r\n[00:08:10] ./dmlc-core/include/dmlc/logging.h:208: [00:08:10] ./mshadow/mshadow/./tensor_gpu-inl.h:44: Check failed: e == cudaSuccess CUDA: out of memory\r\n[00:08:10] ./dmlc-core/include/dmlc/logging.h:208: [00:08:10] src/engine/./threaded_engine.h:295: [00:08:10] ./mshadow/mshadow/./tensor_gpu-inl.h:44: Check failed: e == cudaSuccess CUDA: out of memory\r\nterminate called after throwing an instance of 'dmlc::Error'\r\n  what():  [00:08:10] src/engine/./threaded_engine.h:295: [00:08:10] ./mshadow/mshadow/./tensor_gpu-inl.h:44: Check failed: e == cudaSuccess CUDA: out of memory\r\nAborted (core dumped)\r\n```\r\n\r\nIf i load with batch_size = 1, then everything goes correct:\r\n```\r\n>>> model = mx.model.FeedForward.load('/root/models/mxnet_vgg_face', 1, numpy_batch_size=1, ctx=[mx.gpu(0)])  \r\n>>> dataiter = mx.io.ImageRecordIter(path_imgrec=\"/root/test_images/output.bin\", data_shape=(3,224,224), path_imglist=\"/root/test_images/list.txt\", batch_size = 1)\r\n[00:08:43] src/io/iter_image_recordio.cc:67: Loaded ImageList from /root/test_images/list.txt 11 Image records\r\n[00:08:43] src/io/iter_image_recordio.cc:212: ImageRecordIOParser: /root/test_images/output.bin, use 3 threads for decoding..\r\n>>> t1=time.time(); res = model.predict(dataiter); print 'Timing (per image): %sms' % str(((time.time()-t1)*1000))\r\nTiming (per image): 2753.91817093ms\r\n>>> \r\n```\r\nAnd my GPU memory at half it seems:\r\n```\r\nroot@gpu:~/mxnet# watch nvidia-smi\r\nEvery 2,0s: nvidia-smi                                                                                                                                                                                              Sun Nov 22 00:13:50 2015\r\n\r\nSun Nov 22 00:13:50 2015\r\n+------------------------------------------------------+\r\n| NVIDIA-SMI 352.39     Driver Version: 352.39         |\r\n|-------------------------------+----------------------+----------------------+\r\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n|===============================+======================+======================|\r\n|   0  GeForce GTX 650     Off  | 0000:01:00.0     N/A |                  N/A |\r\n| 40%   37C    P8    N/A /  N/A |    801MiB /  2046MiB |     N/A      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n```\r\nMy output.bin is 87Kb:\r\n```\r\n-rw-r--r--  1 root root 87368 ________. 21 19:37 output.bin\r\n```\r\n\r\nSo what is wrong with batch size = 2? Is it loading twice the model if i double the batch size? Or there is an issue with total files of 11, which gives 5.5 if i divide 11 on 2?\r\n\r\nSorry if a dupe, i wasnt able to find an asnwer on such case."},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/158704044","html_url":"https://github.com/dmlc/mxnet/issues/675#issuecomment-158704044","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/675","id":158704044,"user":{"login":"antinucleon","id":1258974,"avatar_url":"https://avatars.githubusercontent.com/u/1258974?v=3","gravatar_id":"","url":"https://api.github.com/users/antinucleon","html_url":"https://github.com/antinucleon","followers_url":"https://api.github.com/users/antinucleon/followers","following_url":"https://api.github.com/users/antinucleon/following{/other_user}","gists_url":"https://api.github.com/users/antinucleon/gists{/gist_id}","starred_url":"https://api.github.com/users/antinucleon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antinucleon/subscriptions","organizations_url":"https://api.github.com/users/antinucleon/orgs","repos_url":"https://api.github.com/users/antinucleon/repos","events_url":"https://api.github.com/users/antinucleon/events{/privacy}","received_events_url":"https://api.github.com/users/antinucleon/received_events","type":"User","site_admin":false},"created_at":"2015-11-22T03:13:29Z","updated_at":"2015-11-22T03:13:29Z","body":"You can try this to see whether it works. Add explicit parameter\n```workspace``` in each convolution layer. Default value is 512MB and it\nmay be too large for ancient cards. Try to set 64MB in each convolution\nlayer, for example:\n```conv2 = mx.sym.Convolution(data=conv1, kernel=(3,3), pad=(1,1),\nworkspace=64, num_filter=128)```\n\nOn Sat, Nov 21, 2015 at 3:02 PM, Tianqi Chen <notifications@github.com>\nwrote:\n\n> Could due to nonlinear memory request of CuDNN, but we are not yet sure\n> about this in the moment.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/dmlc/mxnet/issues/675#issuecomment-158685213>.\n>\n"}}