{"action":"created","issue":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/468","labels_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/468/labels{/name}","comments_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/468/comments","events_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/468/events","html_url":"https://github.com/Lasagne/Lasagne/issues/468","id":111880832,"number":468,"title":"Glorot gains for Leaky_rectify and very_leaky_rectify","user":{"login":"skaae","id":2623134,"avatar_url":"https://avatars.githubusercontent.com/u/2623134?v=3","gravatar_id":"","url":"https://api.github.com/users/skaae","html_url":"https://github.com/skaae","followers_url":"https://api.github.com/users/skaae/followers","following_url":"https://api.github.com/users/skaae/following{/other_user}","gists_url":"https://api.github.com/users/skaae/gists{/gist_id}","starred_url":"https://api.github.com/users/skaae/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/skaae/subscriptions","organizations_url":"https://api.github.com/users/skaae/orgs","repos_url":"https://api.github.com/users/skaae/repos","events_url":"https://api.github.com/users/skaae/events{/privacy}","received_events_url":"https://api.github.com/users/skaae/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-10-16T18:02:20Z","updated_at":"2015-10-19T13:26:14Z","closed_at":"2015-10-17T10:25:38Z","body":"Should we add the correct Glorot gains for `leaky_rectify` and `very_leaky_rectify`. That would be\r\n\r\n```Python\r\nif gain == 'rectify':\r\n    gain = np.sqrt(2)\r\nelif nonlin == 'leaky_rectify':\r\n    gain = np.sqrt(2/(1+0.01**2))\r\nelif nonlin == 'very_leaky_rectify':\r\n    gain = np.sqrt(2/(1+(1/3)**2))\r\n```\r\n\r\nMaybe also add a note to the doc that the correct  init for `leaky_rectify` with `alpha` leakiness is `np.sqrt(2/(1+alpha**2))` "},"comment":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/comments/149212511","html_url":"https://github.com/Lasagne/Lasagne/issues/468#issuecomment-149212511","issue_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/468","id":149212511,"user":{"login":"f0k","id":629706,"avatar_url":"https://avatars.githubusercontent.com/u/629706?v=3","gravatar_id":"","url":"https://api.github.com/users/f0k","html_url":"https://github.com/f0k","followers_url":"https://api.github.com/users/f0k/followers","following_url":"https://api.github.com/users/f0k/following{/other_user}","gists_url":"https://api.github.com/users/f0k/gists{/gist_id}","starred_url":"https://api.github.com/users/f0k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f0k/subscriptions","organizations_url":"https://api.github.com/users/f0k/orgs","repos_url":"https://api.github.com/users/f0k/repos","events_url":"https://api.github.com/users/f0k/events{/privacy}","received_events_url":"https://api.github.com/users/f0k/received_events","type":"User","site_admin":false},"created_at":"2015-10-19T13:26:14Z","updated_at":"2015-10-19T13:26:14Z","body":"> We can also mention it in the nonlinearities docs?\r\n\r\nI think it just belongs in the `init` docs (to every initializer that has a `gain` argument). Adding a helper function would avoid the need of duplicating the recommendations in every docstring (1.0 for linear and sigmoid, 2.0 for tanh (if I remember correctly), sqrt(2) for relu, sqrt(2 / (1 + leakiness**2)) for leaky relu), but it's not so bad repeating the docstring for now.\r\n\r\nHow did you derive the `sqrt(2 / (1 + leakiness**2))` -- does it hold for both `Glorot` and `He`? `He` only takes into account the forward pass, `Glorot` takes the average of the forward and backward pass. Ah, wait, it's Eq. (15) in \"Delving Deep into Rectifiers\", and they already show it's the same for forward and backward. Good then. Please go ahead :)"}}