{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/561","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/561/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/561/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/561/events","html_url":"https://github.com/dmlc/mxnet/issues/561","id":116668604,"number":561,"title":"[Discussion] Uneven load for different GPU device?","user":{"login":"XericZephyr","id":3773185,"avatar_url":"https://avatars.githubusercontent.com/u/3773185?v=3","gravatar_id":"","url":"https://api.github.com/users/XericZephyr","html_url":"https://github.com/XericZephyr","followers_url":"https://api.github.com/users/XericZephyr/followers","following_url":"https://api.github.com/users/XericZephyr/following{/other_user}","gists_url":"https://api.github.com/users/XericZephyr/gists{/gist_id}","starred_url":"https://api.github.com/users/XericZephyr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/XericZephyr/subscriptions","organizations_url":"https://api.github.com/users/XericZephyr/orgs","repos_url":"https://api.github.com/users/XericZephyr/repos","events_url":"https://api.github.com/users/XericZephyr/events{/privacy}","received_events_url":"https://api.github.com/users/XericZephyr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-11-12T23:49:01Z","updated_at":"2015-11-13T00:15:11Z","closed_at":null,"body":"The current set up for mxnet is assuming Multiple GPUs have same or similar capacity. So the data parallelism is done evenly for every GPU. \r\n\r\nNow let's consider, what if we have three GPUs, e.g. one 980 with 3GB memory and two Titan X with 12 GB memory? \r\n\r\nSeems we have two options:\r\n\r\n1. The batch size should be set to fit the 980 device, which wastes the power of the two Titan X. \r\n2. Or we ignore the 980 device and use just two Titan X devices, which wastes the computing power of the 980. \r\n\r\nNeither of them is perfect, so probably we shall figure out a way to distribute the data unevenly to different devices. Just a suggestion, thanks. "},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/156277517","html_url":"https://github.com/dmlc/mxnet/issues/561#issuecomment-156277517","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/561","id":156277517,"user":{"login":"XericZephyr","id":3773185,"avatar_url":"https://avatars.githubusercontent.com/u/3773185?v=3","gravatar_id":"","url":"https://api.github.com/users/XericZephyr","html_url":"https://github.com/XericZephyr","followers_url":"https://api.github.com/users/XericZephyr/followers","following_url":"https://api.github.com/users/XericZephyr/following{/other_user}","gists_url":"https://api.github.com/users/XericZephyr/gists{/gist_id}","starred_url":"https://api.github.com/users/XericZephyr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/XericZephyr/subscriptions","organizations_url":"https://api.github.com/users/XericZephyr/orgs","repos_url":"https://api.github.com/users/XericZephyr/repos","events_url":"https://api.github.com/users/XericZephyr/events{/privacy}","received_events_url":"https://api.github.com/users/XericZephyr/received_events","type":"User","site_admin":false},"created_at":"2015-11-13T00:15:11Z","updated_at":"2015-11-13T00:15:11Z","body":"@mli  Thanks! I am aware of the code. However, only modifying this part is not general enough. I will try to open a PR shortly to solve this issue. Thanks.  "}}