{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/566","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/566/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/566/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/566/events","html_url":"https://github.com/dmlc/mxnet/pull/566","id":116696642,"number":566,"title":"Add uneven load training feature","user":{"login":"XericZephyr","id":3773185,"avatar_url":"https://avatars.githubusercontent.com/u/3773185?v=3","gravatar_id":"","url":"https://api.github.com/users/XericZephyr","html_url":"https://github.com/XericZephyr","followers_url":"https://api.github.com/users/XericZephyr/followers","following_url":"https://api.github.com/users/XericZephyr/following{/other_user}","gists_url":"https://api.github.com/users/XericZephyr/gists{/gist_id}","starred_url":"https://api.github.com/users/XericZephyr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/XericZephyr/subscriptions","organizations_url":"https://api.github.com/users/XericZephyr/orgs","repos_url":"https://api.github.com/users/XericZephyr/repos","events_url":"https://api.github.com/users/XericZephyr/events{/privacy}","received_events_url":"https://api.github.com/users/XericZephyr/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-11-13T04:08:59Z","updated_at":"2015-11-14T01:15:26Z","closed_at":"2015-11-14T01:12:37Z","pull_request":{"url":"https://api.github.com/repos/dmlc/mxnet/pulls/566","html_url":"https://github.com/dmlc/mxnet/pull/566","diff_url":"https://github.com/dmlc/mxnet/pull/566.diff","patch_url":"https://github.com/dmlc/mxnet/pull/566.patch"},"body":"I've created a uneven load training mechanism for multiple devices in a single machine. \r\n\r\n## Usage \r\n\r\nHere is my test code modified from `example/imagenet/inception.py`\r\n```python\r\nwork_load_list = [(0, 5), (1, 1.5), (2, 5)]\r\ngpus = [mx.gpu(i, work_load) for i, work_load in work_load_list]\r\n\r\nmodel = mx.model.FeedForward(ctx=gpus, symbol=softmax, num_epoch=num_epoch,\r\n                             learning_rate=0.05, momentum=0.9, wd=0.00001)\r\n\r\nmodel.fit(X=train, eval_data=val,\r\n          eval_metric=\"acc\",\r\n          batch_end_callback=mx.callback.Speedometer(batch_size),\r\n          epoch_end_callback=mx.callback.do_checkpoint(model_prefix))\r\n```\r\n\r\nHere, `work_load_list` is a list of `(device_index, work_load)` tuple. Pass `gpus` to the `ctx` argument for `model.fit` API could automatically unevenly distribute samples to different devices. \r\n\r\n**If you don't want this feature, you're good to use your old code with this new commit.**\r\n\r\n## Result\r\n\r\nMy test machine is a workstation equipped with a Quadro K4000 and two Tesla K40c. Training with the modified `example/imagenet/inception.py`, with ratio `10:10:3` for `K40c:K40c:K4000`, the training speedometer is shown as: \r\n\r\n```\r\nINFO:root:Iter[0] Batch [100]\tSpeed: 92.63 samples/sec\r\nINFO:root:Iter[0] Batch [150]\tSpeed: 91.22 samples/sec\r\nINFO:root:Iter[0] Batch [200]\tSpeed: 92.20 samples/sec\r\nINFO:root:Iter[0] Batch [250]\tSpeed: 92.53 samples/sec\r\nINFO:root:Iter[0] Batch [300]\tSpeed: 92.55 samples/sec\r\n```\r\n\r\nWith only two Tesla GPUs, we observe a speed around \r\n\r\n```\r\nINFO:root:Iter[0] Batch [12700]\tSpeed: 80.39 samples/sec\r\nINFO:root:Iter[0] Batch [12750]\tSpeed: 80.27 samples/sec\r\nINFO:root:Iter[0] Batch [12800]\tSpeed: 80.31 samples/sec\r\n```\r\n\r\nSo this uneven work load feature is able to squeeze the last GPU on the machine and slightly improve the performance. "},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/156603198","html_url":"https://github.com/dmlc/mxnet/pull/566#issuecomment-156603198","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/566","id":156603198,"user":{"login":"tqchen","id":2577440,"avatar_url":"https://avatars.githubusercontent.com/u/2577440?v=3","gravatar_id":"","url":"https://api.github.com/users/tqchen","html_url":"https://github.com/tqchen","followers_url":"https://api.github.com/users/tqchen/followers","following_url":"https://api.github.com/users/tqchen/following{/other_user}","gists_url":"https://api.github.com/users/tqchen/gists{/gist_id}","starred_url":"https://api.github.com/users/tqchen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tqchen/subscriptions","organizations_url":"https://api.github.com/users/tqchen/orgs","repos_url":"https://api.github.com/users/tqchen/repos","events_url":"https://api.github.com/users/tqchen/events{/privacy}","received_events_url":"https://api.github.com/users/tqchen/received_events","type":"User","site_admin":false},"created_at":"2015-11-14T01:15:26Z","updated_at":"2015-11-14T01:15:26Z","body":"Thanks for the contribution, this is now merged, have added your name to https://github.com/dmlc/mxnet/blob/master/CONTRIBUTORS.md , please feel free to update your information or contribution more to mxnet!"}}