{"action":"created","issue":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/141","labels_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/141/labels{/name}","comments_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/141/comments","events_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/141/events","html_url":"https://github.com/Lasagne/Lasagne/issues/141","id":57851982,"number":141,"title":"Batch Normalization","user":{"login":"skaae","id":2623134,"avatar_url":"https://avatars.githubusercontent.com/u/2623134?v=3","gravatar_id":"","url":"https://api.github.com/users/skaae","html_url":"https://github.com/skaae","followers_url":"https://api.github.com/users/skaae/followers","following_url":"https://api.github.com/users/skaae/following{/other_user}","gists_url":"https://api.github.com/users/skaae/gists{/gist_id}","starred_url":"https://api.github.com/users/skaae/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/skaae/subscriptions","organizations_url":"https://api.github.com/users/skaae/orgs","repos_url":"https://api.github.com/users/skaae/repos","events_url":"https://api.github.com/users/skaae/events{/privacy}","received_events_url":"https://api.github.com/users/skaae/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":62,"created_at":"2015-02-16T21:15:29Z","updated_at":"2015-08-11T14:34:33Z","closed_at":null,"body":"Moved from https://github.com/benanne/Lasagne/issues/133"},"comment":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/comments/129907930","html_url":"https://github.com/Lasagne/Lasagne/issues/141#issuecomment-129907930","issue_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/141","id":129907930,"user":{"login":"nouiz","id":180987,"avatar_url":"https://avatars.githubusercontent.com/u/180987?v=3","gravatar_id":"","url":"https://api.github.com/users/nouiz","html_url":"https://github.com/nouiz","followers_url":"https://api.github.com/users/nouiz/followers","following_url":"https://api.github.com/users/nouiz/following{/other_user}","gists_url":"https://api.github.com/users/nouiz/gists{/gist_id}","starred_url":"https://api.github.com/users/nouiz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nouiz/subscriptions","organizations_url":"https://api.github.com/users/nouiz/orgs","repos_url":"https://api.github.com/users/nouiz/repos","events_url":"https://api.github.com/users/nouiz/events{/privacy}","received_events_url":"https://api.github.com/users/nouiz/received_events","type":"User","site_admin":false},"created_at":"2015-08-11T14:34:33Z","updated_at":"2015-08-11T14:34:33Z","body":"Composite support multiple output for a long time. It was some\noptimization(causing graph) and the GpuElemwise(also causing crash) that\ndidn't supported that.\n\nIn fact, the more I think about this, the more I think it is easy to do,\nbut I have other stuff to do today and won't be available until 2-3 weeks.\nA quick implementation copied from OpFromGraph:\n\n    def grad(self, inputs, output_grads):\n        gs = theano.gradient.grad(cost=None,\n                                  known_grads=dict(izip(self.outputs,\n                                                        output_grads)),\n                                  wrt=self.inputs,\n                                  disconnected_inputs='ignore')\n        new_op = Composite(self.inputs + output_grads, gs)\n        return new_op(*(inputs + output_grads))\n\n\n\nIf you wan to try it. The only inconvenince is that it will except all\ninputs of the forward and all the grad signals to be needed for the grad op.\n\nI think this could be worked around with something like this:\n\n    def grad(self, inputs, output_grads):\n        gs = theano.gradient.grad(cost=None,\n                                  known_grads=dict(izip(self.outputs,\n                                                        output_grads)),\n                                  wrt=self.inputs,\n                                  disconnected_inputs='ignore')\n        used_inputs = theano.gof.graph.inputs(gs)\n        new_inputs_inner = []\n        new_inputs_outer = []\n        for in_i, out_i in zip(inputs, self.inputs):# + output_grads:\n           if in_i in used_inputs:\n                new_inputs_inner.append(in_i)\n                new_inputs_outer.append(out_i)\n        for o_grad output_grads:\n           if o_grad in used_inputs:\n                new_inputs_inner.append(o_grad)\n                new_inputs_outer.append(o_grad)\n\n        new_op = Composite(new_inputs_inner, gs)\n        return new_op(*new_inputs_outer)\n\nIt also need the connection pattern, this work with the first version of\ngrad above, but not the second (need update for the inputs I think.\n\n    def connection_pattern(self, node):\n        \"\"\"\n        Return connection pattern of subfgraph defined by inputs and outputs\n        \"\"\"\n        return io_connection_pattern(self.inputs, self.outputs)\n\n\nOn Tue, Aug 11, 2015 at 9:38 AM, Jan Schlüter <notifications@github.com>\nwrote:\n\n> I forgot that the grad wasn't defined. We could implement Composite.grad.\n>\n> But the gradient would need to become a multiple-output elemwise Op again,\n> can Composite.grad do this at all? Maybe it's not worth doing a general\n> implementation of Composite.grad, all this will become unnecessary once\n> Theano introduces the multi-output elemwise Op itself via an optimization.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/Lasagne/Lasagne/issues/141#issuecomment-129876241>.\n>\n"}}