{"action":"created","issue":{"url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/66","labels_url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/66/labels{/name}","comments_url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/66/comments","events_url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/66/events","html_url":"https://github.com/soumith/convnet-benchmarks/issues/66","id":116287942,"number":66,"title":"Benchmark TensorFlow","user":{"login":"soumith","id":1310570,"avatar_url":"https://avatars.githubusercontent.com/u/1310570?v=3","gravatar_id":"","url":"https://api.github.com/users/soumith","html_url":"https://github.com/soumith","followers_url":"https://api.github.com/users/soumith/followers","following_url":"https://api.github.com/users/soumith/following{/other_user}","gists_url":"https://api.github.com/users/soumith/gists{/gist_id}","starred_url":"https://api.github.com/users/soumith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumith/subscriptions","organizations_url":"https://api.github.com/users/soumith/orgs","repos_url":"https://api.github.com/users/soumith/repos","events_url":"https://api.github.com/users/soumith/events{/privacy}","received_events_url":"https://api.github.com/users/soumith/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":55,"created_at":"2015-11-11T08:25:43Z","updated_at":"2015-11-12T14:51:15Z","closed_at":null,"body":"Google's TensorFlow benchmarks are here!\r\n\r\nI've run the benchmarks on the Imagenet Winners. \r\nWhen I saw issues with the numbers, memory etc., I emailed @Yangqing to confirm what I'm seeing, and that it is expected. \r\n\r\nWith that disclaimer out of the way, here's some things that you should know about TensorFlow (as of the pip version that I installed today):\r\n- in-place ReLU seems non-existent in practice.\r\n  - Yangqing says: \"right now there are little in-place operations in TensorFlow and we pretty much rely on the scheduler and the memory pool to allocate and deallocate memory\"\r\n- Supports CuDNN R2. No R3 support yet, Yangqing says the next version they are going to support is likely R4.\r\n\r\nComing to the benchmarks:\r\n- Googlenet with batchsize 128 goes Out of Memory. The largest batch-size I could fit is 16 (tried 16, 32, 64, 128)\r\n- VGG with batchsize 64 goes Out of Memory. The largest batch-size I could fit is 32 (tried 32, 64).\r\n- I've also computed Torch7+CuDNN-R2 baselines for these batch-sizes.\r\n\r\n**[AlexNet (One Weird Trick paper)](https://code.google.com/p/cuda-convnet2/source/browse/layers/layers-imagenet-1gpu.cfg)** - Input 128x3x224x224\r\n\r\n| Library        | Time (ms)  | forward (ms) | backward (ms) |\r\n|:------------------------:| ----------:| ------------:| -------------:|\r\n| CuDNN-R3 (Torch)      |      96    |  32          |   64          |\r\n| Nervana (Neon)    |      101   |  32          |    69         |\r\n| CuDNN-R2 (Torch)  |      231   |  70          |   161         |\r\n| **TensorFlow**              |      326   |  96          |  230         |\r\n\r\n**[Overfeat [fast]](http://arxiv.org/abs/1312.6229)** - Input 128x3x231x231\r\n\r\n| Library                 | Time (ms)         | forward (ms)            | backward (ms)            |\r\n|:------------------------:| -----------------:| -----------------------:| ------------------------:|\r\n| CuDNN-R3 (Torch)      |         326       |  113                    |   213                    |\r\n| fbfft (Torch)             |         342       |  114                    |   227                    |\r\n| CuDNN-R2 (Torch)          |         810       |  234                    |   576                    |\r\n| TensorFlow            |      1084         |  316                    |   768                    |\r\n\r\n**[OxfordNet [Model-A]](http://arxiv.org/abs/1409.1556/)** - Input **32**x3x224x224\r\n\r\n| Library                  |  Time (ms)         | forward (ms)            | backward (ms)            |\r\n|:------------------------:| -----------------:| -----------------------:| ------------------------:|\r\n| CuDNN-R2 (Torch)   | 564       |  234                    |   330                    |\r\n| TensorFlow               |  933         |  273                    |   660                    |\r\n\r\n**[GoogleNet V1](http://research.google.com/pubs/pub43022.html)** - Input **16**x3x224x224\r\n\r\n| Library                  |  Time (ms)         | forward (ms)            | backward (ms) |\r\n|:------------------------:| -----------------:| -----------------------:| ------------------------:|\r\n| CuDNN-R2 (Torch)  | 564           |  174                    |   390                    |\r\n| TensorFlow               |  590          |  54                      |   536                    |\r\n\r\n**Note that at batch size of 16, googlenet with CuDNN-R2 + Torch likely runs into dispatching overhead, so it's an exotic comparison, but not practically very interesting or encouraging.**\r\n\r\nThere you go. \r\n\r\nI'm assuming that the first release of TensorFlow is still quite unpolished, and that they will improve it over time with various memory and time optimizations baked in."},"comment":{"url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/comments/156125375","html_url":"https://github.com/soumith/convnet-benchmarks/issues/66#issuecomment-156125375","issue_url":"https://api.github.com/repos/soumith/convnet-benchmarks/issues/66","id":156125375,"user":{"login":"xingdi-eric-yuan","id":4028684,"avatar_url":"https://avatars.githubusercontent.com/u/4028684?v=3","gravatar_id":"","url":"https://api.github.com/users/xingdi-eric-yuan","html_url":"https://github.com/xingdi-eric-yuan","followers_url":"https://api.github.com/users/xingdi-eric-yuan/followers","following_url":"https://api.github.com/users/xingdi-eric-yuan/following{/other_user}","gists_url":"https://api.github.com/users/xingdi-eric-yuan/gists{/gist_id}","starred_url":"https://api.github.com/users/xingdi-eric-yuan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xingdi-eric-yuan/subscriptions","organizations_url":"https://api.github.com/users/xingdi-eric-yuan/orgs","repos_url":"https://api.github.com/users/xingdi-eric-yuan/repos","events_url":"https://api.github.com/users/xingdi-eric-yuan/events{/privacy}","received_events_url":"https://api.github.com/users/xingdi-eric-yuan/received_events","type":"User","site_admin":false},"created_at":"2015-11-12T14:51:15Z","updated_at":"2015-11-12T14:51:15Z","body":"@soumith Thanks you."}}