{"action":"created","issue":{"url":"https://api.github.com/repos/cazala/synaptic/issues/30","labels_url":"https://api.github.com/repos/cazala/synaptic/issues/30/labels{/name}","comments_url":"https://api.github.com/repos/cazala/synaptic/issues/30/comments","events_url":"https://api.github.com/repos/cazala/synaptic/issues/30/events","html_url":"https://github.com/cazala/synaptic/issues/30","id":71865547,"number":30,"title":"Output/Hidden to Hidden gatings in LSTM-RNN","user":{"login":"Sleepwalking","id":4531595,"avatar_url":"https://avatars.githubusercontent.com/u/4531595?v=3","gravatar_id":"","url":"https://api.github.com/users/Sleepwalking","html_url":"https://github.com/Sleepwalking","followers_url":"https://api.github.com/users/Sleepwalking/followers","following_url":"https://api.github.com/users/Sleepwalking/following{/other_user}","gists_url":"https://api.github.com/users/Sleepwalking/gists{/gist_id}","starred_url":"https://api.github.com/users/Sleepwalking/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Sleepwalking/subscriptions","organizations_url":"https://api.github.com/users/Sleepwalking/orgs","repos_url":"https://api.github.com/users/Sleepwalking/repos","events_url":"https://api.github.com/users/Sleepwalking/events{/privacy}","received_events_url":"https://api.github.com/users/Sleepwalking/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":14,"created_at":"2015-04-29T12:42:48Z","updated_at":"2015-04-30T10:10:25Z","closed_at":null,"body":"According to Felix Gers' dissertation[1], gates on memory cells have connection not only from input layer, but also the memory cells themselves. However, Architect currently only projects input-to-out/forget/in gates connections for LSTMs (except peepholes),\r\n\r\n```javascript\r\n      inputLayer.project(inputGate);\r\n      inputLayer.project(forgetGate);\r\n      inputLayer.project(outputGate);\r\n```\r\n\r\nwhich means that the neural network remembers/forgets information **only based on its current inputs**, which could be diastrous for certain tasks that require long term memory.\r\nIn some other applications memory cells are even gated by outputs. Besides gating, first order connections from output to hidden layers also exist in some literatures.\r\n\r\nThis observation provides an insight for why the [Wikipedia language modeling task](http://synaptic.juancazala.com/#/wikipedia) doesn't give promising results even after hours of training. Through informal test enabling hidden layer to gates connections, the network is able to reproduce text such as \"the of the of the of the of the of ...\" on its own. I also trained a LSTM with 70 memory cells on some short paragraphs and the network can exactly reproduce two or three sentences on its own.\r\n\r\nI'm going to run further tests to compare the hidden-to-gates connected LSTMs with input-to-gates connected ones.\r\n\r\n[1] Gers, Felix. *Long short-term memory in recurrent neural networks*. PhD dissertation, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland, 2001.\r\n"},"comment":{"url":"https://api.github.com/repos/cazala/synaptic/issues/comments/97727708","html_url":"https://github.com/cazala/synaptic/issues/30#issuecomment-97727708","issue_url":"https://api.github.com/repos/cazala/synaptic/issues/30","id":97727708,"user":{"login":"Sleepwalking","id":4531595,"avatar_url":"https://avatars.githubusercontent.com/u/4531595?v=3","gravatar_id":"","url":"https://api.github.com/users/Sleepwalking","html_url":"https://github.com/Sleepwalking","followers_url":"https://api.github.com/users/Sleepwalking/followers","following_url":"https://api.github.com/users/Sleepwalking/following{/other_user}","gists_url":"https://api.github.com/users/Sleepwalking/gists{/gist_id}","starred_url":"https://api.github.com/users/Sleepwalking/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Sleepwalking/subscriptions","organizations_url":"https://api.github.com/users/Sleepwalking/orgs","repos_url":"https://api.github.com/users/Sleepwalking/repos","events_url":"https://api.github.com/users/Sleepwalking/events{/privacy}","received_events_url":"https://api.github.com/users/Sleepwalking/received_events","type":"User","site_admin":false},"created_at":"2015-04-30T10:10:25Z","updated_at":"2015-04-30T10:10:25Z","body":"> Sometimes the MSE converges within 100 epochs while some other times it doesn't converge after 500 epochs.\r\n\r\nThis is because 15 hidden units are too many for our simple timing task. I tried again with a 2-5-1 topology and here are the new results.\r\n\r\n[A bug](https://github.com/cazala/synaptic/issues/30#ref-pullrequest-72088088) in forward propagation was fixed before this test.\r\nThe learning rate is set to 0.1 and decays 0.99 times each epoch. Other settings remain same.\r\n\r\nLSTM without H2G\r\n---\r\n![3rd](https://cloud.githubusercontent.com/assets/4531595/7410642/5f1829b2-ef63-11e4-8a5c-2af12ae8f843.png)\r\n\r\n**Output Sample**\r\n![3rd-sample](https://cloud.githubusercontent.com/assets/4531595/7410643/66219f9a-ef63-11e4-866b-13150a6ec2bb.png)\r\n\r\nLSTM with H2G\r\n---\r\n![4th](https://cloud.githubusercontent.com/assets/4531595/7410647/71801c54-ef63-11e4-8eeb-4944b489eccc.png)\r\n\r\n**Output Sample**\r\n![4th-sample](https://cloud.githubusercontent.com/assets/4531595/7410651/75ccfa2a-ef63-11e4-8d12-43da85ab3723.png)\r\n\r\nComments\r\n---\r\n* LSTM with H2G (MSE = 0.001467) performs significantly better than LSTM without H2G (MSE = 0.006556);\r\n* Large networks easily end up in local minimum during training. That's why 2-5-1 can outperform 2-15-1;\r\n* C code for LSTM with H2G is 28.8KB; C code for LSTM without H2G is 17.7KB. The difference in size becomes smaller for small networks and the comparison is fairer - in fact, the 2-5-1 LSTM with H2G achieves the best-ever result on our timing task.\r\n"}}