{"action":"created","issue":{"url":"https://api.github.com/repos/BVLC/caffe/issues/2903","labels_url":"https://api.github.com/repos/BVLC/caffe/issues/2903/labels{/name}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/2903/comments","events_url":"https://api.github.com/repos/BVLC/caffe/issues/2903/events","html_url":"https://github.com/BVLC/caffe/pull/2903","id":100257887,"number":2903,"title":"Multi-GPU Data Parallelism (with Parallel Data Layers)","user":{"login":"ronghanghu","id":6997335,"avatar_url":"https://avatars.githubusercontent.com/u/6997335?v=3","gravatar_id":"","url":"https://api.github.com/users/ronghanghu","html_url":"https://github.com/ronghanghu","followers_url":"https://api.github.com/users/ronghanghu/followers","following_url":"https://api.github.com/users/ronghanghu/following{/other_user}","gists_url":"https://api.github.com/users/ronghanghu/gists{/gist_id}","starred_url":"https://api.github.com/users/ronghanghu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronghanghu/subscriptions","organizations_url":"https://api.github.com/users/ronghanghu/orgs","repos_url":"https://api.github.com/users/ronghanghu/repos","events_url":"https://api.github.com/users/ronghanghu/events{/privacy}","received_events_url":"https://api.github.com/users/ronghanghu/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/BVLC/caffe/labels/focus","name":"focus","color":"006b75"},{"url":"https://api.github.com/repos/BVLC/caffe/labels/ready%20for%20review","name":"ready for review","color":"bfe5bf"},{"url":"https://api.github.com/repos/BVLC/caffe/labels/speed-up","name":"speed-up","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":47,"created_at":"2015-08-11T08:34:26Z","updated_at":"2015-08-13T20:10:41Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/BVLC/caffe/pulls/2903","html_url":"https://github.com/BVLC/caffe/pull/2903","diff_url":"https://github.com/BVLC/caffe/pull/2903.diff","patch_url":"https://github.com/BVLC/caffe/pull/2903.patch"},"body":"This is my package of #2870 (and originally, #2114)\r\n\r\nModification: Allow data layers (and also PythonLayer when used as data layer) to be shared among worker solver's training net, and also test net for future-proof if one wants to do Multi-GPU testing. Data layers are locked during forward to ensure sequential forward. Now all worker solvers fetch data from one single data layer.\r\n\r\nThis ensure that single-gpu training is consistent with multi-gpu training, and allow tests in #2870 to pass. Otherwise in #2870 (#2114) , there are multiple data layers created for worker solver, and these data layers are unaware of each other. This can be a serious issue if one uses deterministic data layers or turn off shuffling. In such case, since data layers in each worker solver reads the same data, one eventually gets same gradient on each solver, so it is almost equivalent to multiply learning rate by GPU number. This is definitely not the desired behavior of Multi-GPU data parallelism, since one should train on different subsets of dataset. Although in #2114 a DataReader is provided, it only applied to leveldb and lmdb, and is hardly extensible to other data layers.\r\n\r\n**DataReader is preserved in this PR and LMDB/LEVELDB DataLayer is not shared.**\r\n\r\n### TODOs\r\n\r\n- [x] Add ShareInParallel function to layer.hpp, data_layer.hpp and pythonlayer.hpp .\r\n- [x] Implement share layers during net construction, construct top blobs of shared layers.\r\n- [x] Add lock to forward in layer.hpp to lock layers.\r\n- [x] Share layers during workersolver construction.\r\n- [x] ~~Remove DataReader. Restore old behavior of DataLayer.~~ DataReader is kept.\r\n- [x] Test `make runtest` on multiple GPU machine.\r\n- [x] Test multi-gpu training on MNIST. (log: https://gist.github.com/ronghanghu/d66d63882c25b31b6148)\r\n- [ ] Test multi-gpu training on ILSVRC.\r\n- [x] Fix NVCC warning on boost/thread.hpp to get Travis CI pass.\r\n\r\n### Drawback\r\n\r\nMulti-GPU training is numerically non-deterministic on data layers excepted for LMDB/LEVELDB DataLayer, see https://github.com/BVLC/caffe/pull/2903#issuecomment-130133266"},"comment":{"url":"https://api.github.com/repos/BVLC/caffe/issues/comments/130825349","html_url":"https://github.com/BVLC/caffe/pull/2903#issuecomment-130825349","issue_url":"https://api.github.com/repos/BVLC/caffe/issues/2903","id":130825349,"user":{"login":"thatguymike","id":3503919,"avatar_url":"https://avatars.githubusercontent.com/u/3503919?v=3","gravatar_id":"","url":"https://api.github.com/users/thatguymike","html_url":"https://github.com/thatguymike","followers_url":"https://api.github.com/users/thatguymike/followers","following_url":"https://api.github.com/users/thatguymike/following{/other_user}","gists_url":"https://api.github.com/users/thatguymike/gists{/gist_id}","starred_url":"https://api.github.com/users/thatguymike/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatguymike/subscriptions","organizations_url":"https://api.github.com/users/thatguymike/orgs","repos_url":"https://api.github.com/users/thatguymike/repos","events_url":"https://api.github.com/users/thatguymike/events{/privacy}","received_events_url":"https://api.github.com/users/thatguymike/received_events","type":"User","site_admin":false},"created_at":"2015-08-13T20:10:41Z","updated_at":"2015-08-13T20:10:41Z","body":"Okay, my 2 intensive tests have passed convergence training.  I think we should merge and then we can get back to perf issues.\r\n\r\n@sguada -  I think we need to look more generally on randomly building batches.  You don't want to do this in LMDB.  I agree as a stop gap, especially if we are going to thread transforms, then we can start at least supporting randomization of the images in the batch in the data layer. "}}