{"action":"created","issue":{"url":"https://api.github.com/repos/cazala/synaptic/issues/30","labels_url":"https://api.github.com/repos/cazala/synaptic/issues/30/labels{/name}","comments_url":"https://api.github.com/repos/cazala/synaptic/issues/30/comments","events_url":"https://api.github.com/repos/cazala/synaptic/issues/30/events","html_url":"https://github.com/cazala/synaptic/issues/30","id":71865547,"number":30,"title":"Output/Hidden to Hidden gatings in LSTM-RNN","user":{"login":"Sleepwalking","id":4531595,"avatar_url":"https://avatars.githubusercontent.com/u/4531595?v=3","gravatar_id":"","url":"https://api.github.com/users/Sleepwalking","html_url":"https://github.com/Sleepwalking","followers_url":"https://api.github.com/users/Sleepwalking/followers","following_url":"https://api.github.com/users/Sleepwalking/following{/other_user}","gists_url":"https://api.github.com/users/Sleepwalking/gists{/gist_id}","starred_url":"https://api.github.com/users/Sleepwalking/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Sleepwalking/subscriptions","organizations_url":"https://api.github.com/users/Sleepwalking/orgs","repos_url":"https://api.github.com/users/Sleepwalking/repos","events_url":"https://api.github.com/users/Sleepwalking/events{/privacy}","received_events_url":"https://api.github.com/users/Sleepwalking/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-04-29T12:42:48Z","updated_at":"2015-04-29T14:45:18Z","closed_at":null,"body":"According to Felix Gers' dissertation[1], memory cells are not only gated by inputs and peepholes, but also by themselves. However, Architect currently only projects input-to-out/forget/in gates connections for LSTMs (except peepholes),\r\n\r\n```javascript\r\n      inputLayer.project(inputGate);\r\n      inputLayer.project(forgetGate);\r\n      inputLayer.project(outputGate);\r\n```\r\n\r\nwhich means that the neural network remembers/forgets information **only based on its current inputs**, which could be diastrous for certain tasks that require long term memory.\r\nIn some other applications memory cells are even gated by outputs. Besides gating, first order connections from output to hidden layers also exist in some literatures.\r\n\r\nThis observation provides an insight for why the [Wikipedia language modeling task](http://synaptic.juancazala.com/#/wikipedia) doesn't give promising results even after hours of training. Through informal test enabling hidden layer to gates connections, the network is able to reproduce text such as \"the of the of the of the of the of ...\" on its own. I also trained a LSTM with 70 memory cells on some short paragraphs and the network can exactly reproduce two or three sentences on its own.\r\n\r\nI'm going to run further tests to compare the hidden-to-gates connected LSTMs with input-to-gates connected ones.\r\n\r\n[1] Gers, Felix. *Long short-term memory in recurrent neural networks*. PhD dissertation, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland, 2001.\r\n"},"comment":{"url":"https://api.github.com/repos/cazala/synaptic/issues/comments/97453318","html_url":"https://github.com/cazala/synaptic/issues/30#issuecomment-97453318","issue_url":"https://api.github.com/repos/cazala/synaptic/issues/30","id":97453318,"user":{"login":"Sleepwalking","id":4531595,"avatar_url":"https://avatars.githubusercontent.com/u/4531595?v=3","gravatar_id":"","url":"https://api.github.com/users/Sleepwalking","html_url":"https://github.com/Sleepwalking","followers_url":"https://api.github.com/users/Sleepwalking/followers","following_url":"https://api.github.com/users/Sleepwalking/following{/other_user}","gists_url":"https://api.github.com/users/Sleepwalking/gists{/gist_id}","starred_url":"https://api.github.com/users/Sleepwalking/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Sleepwalking/subscriptions","organizations_url":"https://api.github.com/users/Sleepwalking/orgs","repos_url":"https://api.github.com/users/Sleepwalking/repos","events_url":"https://api.github.com/users/Sleepwalking/events{/privacy}","received_events_url":"https://api.github.com/users/Sleepwalking/received_events","type":"User","site_admin":false},"created_at":"2015-04-29T14:45:18Z","updated_at":"2015-04-29T14:45:18Z","body":"Large networks are trained notoriously slow...\r\nI actually translated the generated codes into C and compiled them using clang; encapsulated into MEX files and trained & tested the network using GNU Octave.\r\nLSTM networks with 30 binary inputs, 72 memory cells, and 30 binary outputs are trained to **predict the next character based on current character** (and its memory).\r\nInput/Output format:\r\n`| 26: a-z | 1:period or comma | 1:space | 1:quote | 1:parentheses |`\r\nThe training set is some super stupid non-sense,\r\n`pata thinks that kata wants the pala of the kaba, because kata hates the pala of the kaba so much. kaba wants to kill pata so that kaba can kill kata. this is so nice.`\r\nspecially designed to see if the network can capture the context because `pata`, `kata`, `kaba` and `pala`... occur multiple times in the text.\r\nTo work around initialization problems, the training set is repeated lots of times and joined together to give a 7000-character-long text,\r\n```\r\npata thinks that kata wants the pala of the kaba, because kata hates the pala of the kaba so \r\nmuch. kaba wants to kill pata so that kaba can kill kata. this is so nice. pata thinks that kata \r\nwants the pala of the kaba, because kata hates the pala of the kaba so much. kaba wants \r\nto kill pata so that kaba can kill kata. this is so nice. pata thinks that kata wants the pala of \r\nthe kaba, because kata hates the pala of the kaba so much. kaba wants to kill pata so that \r\nkaba can kill kata. this is so nice.\r\n...\r\n```\r\nThe LSTM is trained using stochastic gradient descent with learning rate = 0.01.\r\n\r\nResult Without Hidden-to-Gates Connections\r\n---\r\nMSE at epoch 30 = 0.111187\r\nOutput sample:\r\n```\r\nt kill pata thant kill kata so this kata the kaba wans thes o much. kill pata so that kaba wans\r\nto kill pata so that kaba can kill kata this so nice. kata. this inis s nice. pata this s o nice. kill\r\npata that kaba so nthe kaba kaba so chice. kaba so nthice. kata kaba wants thes the pala \r\nof the kaba so much. kaba can kill kata this so inis s nice. pata this so nice. kata thinks that \r\nkata so the kaba wants the pala of the kaba wants to kill kata so that kaba can kill kata so \r\nthis so nice. pata thinks that kata wants the pala of the kaba wants the kaba so much. kaba\r\ncan kill kata so this so nice. pata thinks that kata wants the pala of the kaba wants to kill pata\r\nso tha kaba can this kil so katat kata wants the pala of the kaba wants to kill kata so that kaba\r\ncan kill kata so this so nice. kill pata tha kata wants the kaba wanso this s o nice. pata thinks\r\nthat kata wants the pala of the kaba wants to kill kata so that kaba can kill kata so this so nice.\r\npata thinks that kata wants the pala of the kaba wants the kaba so much.\r\n...\r\n```\r\n\r\nResult Without Hidden-to-Gates Connections\r\n---\r\nMSE at epoch 30 = 0.203307\r\nOutput sample:\r\n```\r\nthis this so thinininise. ts the pata the pala of the kaba ba can kill kata so kila so this so thiso this\r\nso thiso the pata the pala of the kaba ba can kill kata so kila so this so thiso this so thiso the pata\r\nthe pala of the kaba ban cats the kaba wants the kaba so the kaba can kill kata so kata this s ice.\r\npata this this so the paba of the kaba. can kill kata this s ice. pata this the kaba ban kila so ice.\r\nkilll pata that kaba so kanis this so thiso thise so that kata so that kaba can kill kata this s ice.\r\npata this the kaba ban cats the kaba wants the kaba so the kaba canks this the kaba ba can kill\r\nkata so kill so ice. pata that kata wants the kaba wants the kaba of kill kata so this so thiso this\r\nso the paba o cathis the kaba wants the kaba of the kaba wants thes o kaba cants the kaba\r\nwants the kaba so the kaba can kill kata so kata this s ice. pata this this so the paba of the kaba.\r\ncan kill kata this s ice. pata this this so the paba of the kaba ba cants the killats s tha o kaba can\r\nkill so ice. pata this this so thininise ts t ...\r\n```\r\n\r\nIt's kind of weird that the one with hidden-to-gates connections perform even worse than the original one. This is probably because the training hasn't converged yet. Seems like the learning rate is so high that MSE bounces back after 25 epochs.\r\nNote that the later one is 3 times larger than the one without hidden-to-gates connections; it gives a 23MB C file and training is extremely slow even translated to C!\r\n\r\n\r\n"}}