{"action":"created","issue":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/225","labels_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/225/labels{/name}","comments_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/225/comments","events_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/225/events","html_url":"https://github.com/Lasagne/Lasagne/pull/225","id":71999616,"number":225,"title":"added NonlinearityOnlyLayer for use with MultipleInputLayer","user":{"login":"mheilman","id":181744,"avatar_url":"https://avatars.githubusercontent.com/u/181744?v=3","gravatar_id":"","url":"https://api.github.com/users/mheilman","html_url":"https://github.com/mheilman","followers_url":"https://api.github.com/users/mheilman/followers","following_url":"https://api.github.com/users/mheilman/following{/other_user}","gists_url":"https://api.github.com/users/mheilman/gists{/gist_id}","starred_url":"https://api.github.com/users/mheilman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mheilman/subscriptions","organizations_url":"https://api.github.com/users/mheilman/orgs","repos_url":"https://api.github.com/users/mheilman/repos","events_url":"https://api.github.com/users/mheilman/events{/privacy}","received_events_url":"https://api.github.com/users/mheilman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-04-29T21:48:09Z","updated_at":"2015-04-30T16:12:52Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Lasagne/Lasagne/pulls/225","html_url":"https://github.com/Lasagne/Lasagne/pull/225","diff_url":"https://github.com/Lasagne/Lasagne/pull/225.diff","patch_url":"https://github.com/Lasagne/Lasagne/pull/225.patch"},"body":"This adds a layer class that just performs a nonlinearity.\r\nThis is for use in conjunction with MultipleInputsLayer subclasses, which just add or concatenate outputs from layers.  For example, one might want to take inputs of length n and m, perform linear transformations on them using DenseLayer objects, then add the transformed outputs with an ElemwiseSumLayer, and then use this class to apply a nonlinearity (note that this would be different from applying the nonlinearity and then adding).  Alternatively, one could use a ConcatLayer followed by a DenseLayer, but concatenation is less efficient than adding."},"comment":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/comments/97861765","html_url":"https://github.com/Lasagne/Lasagne/pull/225#issuecomment-97861765","issue_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/225","id":97861765,"user":{"login":"f0k","id":629706,"avatar_url":"https://avatars.githubusercontent.com/u/629706?v=3","gravatar_id":"","url":"https://api.github.com/users/f0k","html_url":"https://github.com/f0k","followers_url":"https://api.github.com/users/f0k/followers","following_url":"https://api.github.com/users/f0k/following{/other_user}","gists_url":"https://api.github.com/users/f0k/gists{/gist_id}","starred_url":"https://api.github.com/users/f0k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f0k/subscriptions","organizations_url":"https://api.github.com/users/f0k/orgs","repos_url":"https://api.github.com/users/f0k/repos","events_url":"https://api.github.com/users/f0k/events{/privacy}","received_events_url":"https://api.github.com/users/f0k/received_events","type":"User","site_admin":false},"created_at":"2015-04-30T16:12:52Z","updated_at":"2015-04-30T16:12:52Z","body":"> I'd go for `NonlinearityLayer` or even `NonlinLayer` instead. The \"only\" only makes sense when comparing it to `DenseLayer` or `Conv*DLayer`, and I don't think it should be seen as a \"slimmed down\" version of those layers.\r\n\r\nOn the other hand, `NonlinearityLayer` sounds as if it was a specialty of this layer to sport a nonlinearity. I think it's okay to regard `NonlinearityOnlyLayer` as a stripped-down version of usual layers that do something *besides* applying a nonlinearity.\r\n\r\nBy the way, if `nonlinearity` would default to `None` instead of the linear rectifier, we could also just add that keyword to every layer for convenience... :)\r\n\r\nAnyway, I don't have a very strong opinion on the name, as much as I love name discussions.\r\n\r\n---\r\n\r\n> I ran the following script with those flags on an EC2 GPU instance, once for summing, once for concatenation\r\n\r\nCool. The interesting parts are for summing:\r\n>```\r\n21.0%    84.4%       1.100s       1.10e-03s   1000     4   GpuDot22(GpuFromHost.0, W)\r\n 9.3%    93.7%       0.487s       4.87e-04s   1000     5   GpuGemm{inplace}(GpuDot22.0, TensorConstant{1.0}, GpuFromHost.0, W, TensorConstant{1.0})\r\n```\r\n\r\nAnd for concatenation:\r\n>```\r\n21.0%    75.3%       1.285s       1.29e-03s   1000     4   GpuDot22(GpuJoin.0, W)\r\n20.4%    95.6%       1.243s       1.24e-03s   1000     3   GpuJoin(TensorConstant{1}, GpuFromHost.0, GpuFromHost.0)\r\n```\r\n\r\nYou can nicely see that the sum gets compiled into a dot product and then a gemm call that adds its output to the output of the previous dot product, taking 1.6s in total, while the concatenation and single dot product takes 2.5s in total. Good to know!"}}