{"action":"opened","issue":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/520","labels_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/520/labels{/name}","comments_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/520/comments","events_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/520/events","html_url":"https://github.com/Lasagne/Lasagne/issues/520","id":119927673,"number":520,"title":"Relu weight init doc problem?","user":{"login":"colinfang","id":1499555,"avatar_url":"https://avatars.githubusercontent.com/u/1499555?v=3","gravatar_id":"","url":"https://api.github.com/users/colinfang","html_url":"https://github.com/colinfang","followers_url":"https://api.github.com/users/colinfang/followers","following_url":"https://api.github.com/users/colinfang/following{/other_user}","gists_url":"https://api.github.com/users/colinfang/gists{/gist_id}","starred_url":"https://api.github.com/users/colinfang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/colinfang/subscriptions","organizations_url":"https://api.github.com/users/colinfang/orgs","repos_url":"https://api.github.com/users/colinfang/repos","events_url":"https://api.github.com/users/colinfang/events{/privacy}","received_events_url":"https://api.github.com/users/colinfang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-12-02T12:06:22Z","updated_at":"2015-12-02T12:06:22Z","closed_at":null,"body":"From the [doc](http://lasagne.readthedocs.org/en/latest/modules/init.html#lasagne.init.Glorot), [source](https://github.com/Lasagne/Lasagne/blob/master/lasagne/init.py#L103), it says:\r\n\r\n```{python}\r\nFor a :class:`DenseLayer <lasagne.layers.DenseLayer>`, if ``gain='relu'``\r\n    and ``initializer=Uniform``, the weights are initialized as\r\n    .. math::\r\n       a &= \\\\sqrt{\\\\frac{6}{fan_{in}+fan_{out}}}\\\\\\\\\r\n       W &\\sim U[-a, a]\r\n```\r\nBut from what I read from the souce, the formula is true for a sigmoid activator. In case of Relu, it should be `U[-sqrt(2)a, sqrt(2)a]`, as the gain is sqrt(2).\r\n\r\nAm I right?\r\n\r\n"}}