{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/29","labels_url":"https://api.github.com/repos/fchollet/keras/issues/29/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/29/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/29/events","html_url":"https://github.com/fchollet/keras/pull/29","id":65913624,"number":29,"title":"Add new initializations","user":{"login":"untom","id":3627551,"avatar_url":"https://avatars.githubusercontent.com/u/3627551?v=3","gravatar_id":"","url":"https://api.github.com/users/untom","html_url":"https://github.com/untom","followers_url":"https://api.github.com/users/untom/followers","following_url":"https://api.github.com/users/untom/following{/other_user}","gists_url":"https://api.github.com/users/untom/gists{/gist_id}","starred_url":"https://api.github.com/users/untom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/untom/subscriptions","organizations_url":"https://api.github.com/users/untom/orgs","repos_url":"https://api.github.com/users/untom/repos","events_url":"https://api.github.com/users/untom/events{/privacy}","received_events_url":"https://api.github.com/users/untom/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-04-02T12:13:15Z","updated_at":"2015-04-02T17:27:05Z","closed_at":"2015-04-02T17:27:05Z","pull_request":{"url":"https://api.github.com/repos/fchollet/keras/pulls/29","html_url":"https://github.com/fchollet/keras/pull/29","diff_url":"https://github.com/fchollet/keras/pull/29.diff","patch_url":"https://github.com/fchollet/keras/pull/29.patch"},"body":"This PR adds two new initialization methods:\r\n\r\n*  The method from \"Understanding the difficulty of training deep feedforward neural networks\", Glorot & Bengio, AISTATS 2010. This works very well for sigmoid/tanh activation functions. This is known as the \"xavier\" initialization in [Caffe](http://caffe.berkeleyvision.org/), where it is the default initialization method for dense layers.\r\n\r\n* The improvements on the above method made by [He et. al](http://arxiv.org/abs/1502.01852), which they showed works much better when using ReLU activation functions. \r\n"},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/88980807","html_url":"https://github.com/fchollet/keras/pull/29#issuecomment-88980807","issue_url":"https://api.github.com/repos/fchollet/keras/issues/29","id":88980807,"user":{"login":"fchollet","id":710255,"avatar_url":"https://avatars.githubusercontent.com/u/710255?v=3","gravatar_id":"","url":"https://api.github.com/users/fchollet","html_url":"https://github.com/fchollet","followers_url":"https://api.github.com/users/fchollet/followers","following_url":"https://api.github.com/users/fchollet/following{/other_user}","gists_url":"https://api.github.com/users/fchollet/gists{/gist_id}","starred_url":"https://api.github.com/users/fchollet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fchollet/subscriptions","organizations_url":"https://api.github.com/users/fchollet/orgs","repos_url":"https://api.github.com/users/fchollet/repos","events_url":"https://api.github.com/users/fchollet/events{/privacy}","received_events_url":"https://api.github.com/users/fchollet/received_events","type":"User","site_admin":false},"created_at":"2015-04-02T17:27:05Z","updated_at":"2015-04-02T17:27:05Z","body":"A few things:\r\n\r\n- Use float divisions for Python 2.7 compatibility\r\n- Activations can potentially be used anywhere, on any input shape, so should not be dependent on the number of dimensions of the input.\r\n- I believe the scheme proposed by He is sqrt(2/n_l) where n_l is the number of elements in the layer (n_l = k*k*c, with k*k the dimensions of the 2D input and c the number of channels)."}}