{"action":"created","issue":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/531","labels_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/531/labels{/name}","comments_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/531/comments","events_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/531/events","html_url":"https://github.com/Lasagne/Lasagne/issues/531","id":121675163,"number":531,"title":"Residual learning","user":{"login":"kshmelkov","id":10819534,"avatar_url":"https://avatars.githubusercontent.com/u/10819534?v=3","gravatar_id":"","url":"https://api.github.com/users/kshmelkov","html_url":"https://github.com/kshmelkov","followers_url":"https://api.github.com/users/kshmelkov/followers","following_url":"https://api.github.com/users/kshmelkov/following{/other_user}","gists_url":"https://api.github.com/users/kshmelkov/gists{/gist_id}","starred_url":"https://api.github.com/users/kshmelkov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kshmelkov/subscriptions","organizations_url":"https://api.github.com/users/kshmelkov/orgs","repos_url":"https://api.github.com/users/kshmelkov/repos","events_url":"https://api.github.com/users/kshmelkov/events{/privacy}","received_events_url":"https://api.github.com/users/kshmelkov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2015-12-11T09:56:14Z","updated_at":"2015-12-16T14:43:00Z","closed_at":null,"body":"I have just read MSR report on winning ImageNet 152-layers network: http://arxiv.org/abs/1512.03385 I suggest to discuss how to implement residual blocks flexibly in Lasagne. For those who don't have time to read the paper I just write down the key idea: `y = ReLU(BN(F(x) + x))` where `F(x)` is a couple of conv layers (in paper it is `F(x) = conv(ReLU(BN(conv(x)))`). If dimensions of `x` and `F(x)` don't match, just project `x` linearly.\r\n\r\nDo we have a flexible way to manage subnetworks?"},"comment":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/comments/165129528","html_url":"https://github.com/Lasagne/Lasagne/issues/531#issuecomment-165129528","issue_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/531","id":165129528,"user":{"login":"f0k","id":629706,"avatar_url":"https://avatars.githubusercontent.com/u/629706?v=3","gravatar_id":"","url":"https://api.github.com/users/f0k","html_url":"https://github.com/f0k","followers_url":"https://api.github.com/users/f0k/followers","following_url":"https://api.github.com/users/f0k/following{/other_user}","gists_url":"https://api.github.com/users/f0k/gists{/gist_id}","starred_url":"https://api.github.com/users/f0k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f0k/subscriptions","organizations_url":"https://api.github.com/users/f0k/orgs","repos_url":"https://api.github.com/users/f0k/repos","events_url":"https://api.github.com/users/f0k/events{/privacy}","received_events_url":"https://api.github.com/users/f0k/received_events","type":"User","site_admin":false},"created_at":"2015-12-16T14:43:00Z","updated_at":"2015-12-16T14:43:00Z","body":"> fyi, the example above from @f0k does not work.\r\n\r\nDisclaimer: I just quickly wrote it down after skimming the paper (Figure 3 and the accompanying description), it was never tested.\r\n\r\n> does this look right - this one actually works:\r\n\r\nNo, it doesn't look right, now you're not downsampling at all when increasing the channel count. Did you try the corrected version in https://github.com/Lasagne/Lasagne/issues/531#issuecomment-164776274?\r\n\r\n> Correct me if i am wrong but by reading the article i understood that [...]\r\n\r\nYes, good catch, I hadn't read the full article. Should have mentioned right away that this was just meant to convey the general idea. So this one should be closer to the truth (but still untested, and missing batch normalization):\r\n```python\r\ndef residual_block(layer, num_filters, filter_size=3, stride=1, num_layers=2):\r\n    conv = layer\r\n    if (num_filters != layer.output_shape[1]) or (stride != 1):\r\n        layer = Conv2DLayer(layer, num_filters, filter_size=1, stride=stride, pad=0, nonlinearity=None, b=None)\r\n    for _ in range(num_layers):\r\n        conv = Conv2DLayer(conv, num_filters, filter_size, stride=stride, pad='same')\r\n        stride = 1\r\n    nonlinearity = conv.nonlinearity\r\n    conv.nonlinearity = lasagne.nonlinearities.identity\r\n    return NonlinearityLayer(ElemwiseSumLayer([conv, layer]), nonlinearity)\r\n\r\nlayer = InputLayer(...)\r\nlayer = Conv2DLayer(layer, 64, 7, stride=2, pad='same')\r\nlayer = Pool2DLayer(layer, 2)\r\nfor _ in range(3):\r\n    layer = residual_block(layer, 64)\r\nlayer = residual_block(layer, 128, stride=2)\r\nfor _ in range(3):\r\n    layer = residual_block(layer, 128)\r\nlayer = residual_block(layer, 256, stride=2)\r\nfor _ in range(5):\r\n    layer = residual_block(layer, 256)\r\nlayer = residual_block(layer, 512, stride=2)\r\nfor _ in range(2):\r\n    layer = residual_block(layer, 512)\r\n# avg pool, then fully-connect\r\n```\r\nIf anybody has the time to finish this and replicate some of the experiments in the paper, please do submit a PR to our Lasagne/Recipes collection!"}}