{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/990","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/990/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/990/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/990/events","html_url":"https://github.com/dmlc/mxnet/issues/990","id":123160802,"number":990,"title":"mxnet is slow and use too much CPU when training ResNet","user":{"login":"shiyemin","id":1501158,"avatar_url":"https://avatars.githubusercontent.com/u/1501158?v=3","gravatar_id":"","url":"https://api.github.com/users/shiyemin","html_url":"https://github.com/shiyemin","followers_url":"https://api.github.com/users/shiyemin/followers","following_url":"https://api.github.com/users/shiyemin/following{/other_user}","gists_url":"https://api.github.com/users/shiyemin/gists{/gist_id}","starred_url":"https://api.github.com/users/shiyemin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shiyemin/subscriptions","organizations_url":"https://api.github.com/users/shiyemin/orgs","repos_url":"https://api.github.com/users/shiyemin/repos","events_url":"https://api.github.com/users/shiyemin/events{/privacy}","received_events_url":"https://api.github.com/users/shiyemin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-20T13:51:25Z","updated_at":"2015-12-20T19:24:16Z","closed_at":null,"body":"I'm training ResNet (ImageNet2015, MSRA) on a server with 8*K80. However when using all 16 GPUs (one K80 as two GPUs), all 48 CPUs are utilized. Meanwhile, the mxnet's speed is only 4.5 samples/sec, which is much too slow. Is there any possible reason?"},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/166148504","html_url":"https://github.com/dmlc/mxnet/issues/990#issuecomment-166148504","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/990","id":166148504,"user":{"login":"antinucleon","id":1258974,"avatar_url":"https://avatars.githubusercontent.com/u/1258974?v=3","gravatar_id":"","url":"https://api.github.com/users/antinucleon","html_url":"https://github.com/antinucleon","followers_url":"https://api.github.com/users/antinucleon/followers","following_url":"https://api.github.com/users/antinucleon/following{/other_user}","gists_url":"https://api.github.com/users/antinucleon/gists{/gist_id}","starred_url":"https://api.github.com/users/antinucleon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antinucleon/subscriptions","organizations_url":"https://api.github.com/users/antinucleon/orgs","repos_url":"https://api.github.com/users/antinucleon/repos","events_url":"https://api.github.com/users/antinucleon/events{/privacy}","received_events_url":"https://api.github.com/users/antinucleon/received_events","type":"User","site_admin":false},"created_at":"2015-12-20T19:24:16Z","updated_at":"2015-12-20T19:24:16Z","body":"There can be many reason.\r\n1. How is your network? If network is a bottle neck, it will be slow.\r\n2. What is your batch size? If your batch size is too small, it will be slow.\r\n3. How is your PCI-Bandit? Default kvstore is on CPU, which will copy GPU to CPU then update. If your PCI-Bandit is limited, you can try sync on device. \r\n"}}