{"action":"created","issue":{"url":"https://api.github.com/repos/craffel/nntools/issues/61","labels_url":"https://api.github.com/repos/craffel/nntools/issues/61/labels{/name}","comments_url":"https://api.github.com/repos/craffel/nntools/issues/61/comments","events_url":"https://api.github.com/repos/craffel/nntools/issues/61/events","html_url":"https://github.com/craffel/nntools/issues/61","id":95930437,"number":61,"title":"Matrix multiplication in RecurrentLayer documentation","user":{"login":"tristandeleu","id":2018752,"avatar_url":"https://avatars.githubusercontent.com/u/2018752?v=3","gravatar_id":"","url":"https://api.github.com/users/tristandeleu","html_url":"https://github.com/tristandeleu","followers_url":"https://api.github.com/users/tristandeleu/followers","following_url":"https://api.github.com/users/tristandeleu/following{/other_user}","gists_url":"https://api.github.com/users/tristandeleu/gists{/gist_id}","starred_url":"https://api.github.com/users/tristandeleu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tristandeleu/subscriptions","organizations_url":"https://api.github.com/users/tristandeleu/orgs","repos_url":"https://api.github.com/users/tristandeleu/repos","events_url":"https://api.github.com/users/tristandeleu/events{/privacy}","received_events_url":"https://api.github.com/users/tristandeleu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-07-19T16:49:11Z","updated_at":"2015-07-19T17:26:50Z","closed_at":null,"body":"In the `RecurrentLayer` documentation there is this expression\r\n```\r\nh_t = \\sigma(W_x x_t + W_h h_{t-1} + b)\r\n```\r\nHowever `RecurrentLayer`, which uses `DenseLayer`, works with *post*-multiplications of the weight matrices (and treats `x_t` and `h_t` as row vectors). The actual matrix used to update the hidden state is `W_h^T` -- similarly for `W_x`.\r\n\r\nThere are two possible fixes (which is why I didn't make a PR right away, it all depends on your own preferences):\r\n- Consider `x_t` and `h_t` as column vectors like in the documentation\r\n```\r\nh_t = \\sigma(W_x^T x_t + W_h^T h_{t-1} + b)\r\n```\r\n- Consider `x_t` and `h_t` as row vectors like in the code\r\n```\r\nh_t = \\sigma(x_t W_x + h_{t-1} W_h + b)\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/craffel/nntools/issues/comments/122687249","html_url":"https://github.com/craffel/nntools/issues/61#issuecomment-122687249","issue_url":"https://api.github.com/repos/craffel/nntools/issues/61","id":122687249,"user":{"login":"tristandeleu","id":2018752,"avatar_url":"https://avatars.githubusercontent.com/u/2018752?v=3","gravatar_id":"","url":"https://api.github.com/users/tristandeleu","html_url":"https://github.com/tristandeleu","followers_url":"https://api.github.com/users/tristandeleu/followers","following_url":"https://api.github.com/users/tristandeleu/following{/other_user}","gists_url":"https://api.github.com/users/tristandeleu/gists{/gist_id}","starred_url":"https://api.github.com/users/tristandeleu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tristandeleu/subscriptions","organizations_url":"https://api.github.com/users/tristandeleu/orgs","repos_url":"https://api.github.com/users/tristandeleu/repos","events_url":"https://api.github.com/users/tristandeleu/events{/privacy}","received_events_url":"https://api.github.com/users/tristandeleu/received_events","type":"User","site_admin":false},"created_at":"2015-07-19T17:26:50Z","updated_at":"2015-07-19T17:26:50Z","body":"It is indeed cleaner!\r\nI haven't fully checked `LSTMLayer` and `GRULayer` but it looks like there are similar matrix multiplications issues there as well."}}