{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/620","labels_url":"https://api.github.com/repos/fchollet/keras/issues/620/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/620/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/620/events","html_url":"https://github.com/fchollet/keras/issues/620","id":104169172,"number":620,"title":"[WIP] Recursive container","user":{"login":"EderSantana","id":3902480,"avatar_url":"https://avatars.githubusercontent.com/u/3902480?v=3","gravatar_id":"","url":"https://api.github.com/users/EderSantana","html_url":"https://github.com/EderSantana","followers_url":"https://api.github.com/users/EderSantana/followers","following_url":"https://api.github.com/users/EderSantana/following{/other_user}","gists_url":"https://api.github.com/users/EderSantana/gists{/gist_id}","starred_url":"https://api.github.com/users/EderSantana/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EderSantana/subscriptions","organizations_url":"https://api.github.com/users/EderSantana/orgs","repos_url":"https://api.github.com/users/EderSantana/repos","events_url":"https://api.github.com/users/EderSantana/events{/privacy}","received_events_url":"https://api.github.com/users/EderSantana/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-09-01T02:09:02Z","updated_at":"2015-11-22T23:24:38Z","closed_at":null,"body":"### What?\r\nOver the weekend I worked on a solution to design arbitrary RNNs using Keras API. The result allow us write a vanilla RNN as:\r\n```python\r\nself.input_dim = 2\r\nself.state_dim = 2\r\n\r\nself.model = Recursive(return_sequences=True)\r\n\r\nself.model.add_input('input', ndim=3)  # Input is 3D tensor\r\nself.model.add_state('h', dim=self.state_dim)\r\nself.model.add_node(Dense(self.input_dim, self.state_dim, init='one'), \r\n                    name='i2h', inputs=['input', ])\r\nself.model.add_node(Dense(self.state_dim, self.state_dim, init='orthogonal'),\r\n                    name='h2h', inputs=['h', ])\r\nself.model.add_node(Activation('tanh'). name='rec', inputs=['i2h', 'h2h'], \r\n                    merge_mode='sum', return_state='h', create_output=True)\r\n```\r\nNote that the class definition of SimpleRNN is much bigger than this and we don't have the choice of outputting intermediate values, like ex. the input-to-hidden projection. This should be interesting to design different state based models without having to dig into Theano code. I started the development on the repo I usually put [my Keras extensions](https://github.com/EderSantana/seya/blob/master/seya/layers/containers.py). There is a test [here](https://github.com/EderSantana/seya/blob/master/tests/test_containers.py) showing how to use this new container (there will be lots of printing I used for debugging. That should be cleaned up soon). If there is a general interest on this, I could just PR a new branch here.\r\n\r\n### How? (dev details)\r\nThe Recursive container is basically a Graph container with a few mods. The most important difference is the way we connect layers. Contrary to regular feedforward networks, we cannot use `set_previous` inside the `add_node` method. Everything has to be done inside a `_step` function and we have to take care of the order which we pass arguments to scan (I didn't explore the idea of using dictionaries as inputs to scan yet). In other words, the entire connection logic is moved from `add_node` (like it is done for Sequential and Graph) to `_step`.\r\n\r\n### Next Steps?\r\nThere is a lot of code clean up and refactoring that could possibly make the internals cleaner. For example, in a conversation with @fchollet, he also suggested me that we should define the states as `self.model.add_state('h', dim=self.state_dim, input='rec')` instead of using a `return_state` option inside the `add_node`.\r\n\r\n##### Stateful?\r\nAnother interesting problem is how to handle stateful models, where the hidden states are not wiped out after each batch. In a previous experiment I did, I set up the initial states of an RNN to be a shared variable and defined its update to be the last state returned by scan. I did that inside the `get_output` method. Now that Keras gets all the individual layers `self.updates`, everything else was handled by the Model class. We could also do this here. The problem is that shared variables can't change sizes and we have to make sure we always have batches of the same size, otherwise, we would have to recompile the model. I would love to hear about alternatives for this.\r\n\r\n### Final words\r\nSorry for the long post, but hopefully it will get people interested in developing this API (and/or inspiring new ones) that I believe will make our lives much easier when it comes to design new RNNs."},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/158817941","html_url":"https://github.com/fchollet/keras/issues/620#issuecomment-158817941","issue_url":"https://api.github.com/repos/fchollet/keras/issues/620","id":158817941,"user":{"login":"elanmart","id":10772830,"avatar_url":"https://avatars.githubusercontent.com/u/10772830?v=3","gravatar_id":"","url":"https://api.github.com/users/elanmart","html_url":"https://github.com/elanmart","followers_url":"https://api.github.com/users/elanmart/followers","following_url":"https://api.github.com/users/elanmart/following{/other_user}","gists_url":"https://api.github.com/users/elanmart/gists{/gist_id}","starred_url":"https://api.github.com/users/elanmart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elanmart/subscriptions","organizations_url":"https://api.github.com/users/elanmart/orgs","repos_url":"https://api.github.com/users/elanmart/repos","events_url":"https://api.github.com/users/elanmart/events{/privacy}","received_events_url":"https://api.github.com/users/elanmart/received_events","type":"User","site_admin":false},"created_at":"2015-11-22T23:24:38Z","updated_at":"2015-11-22T23:24:38Z","body":"@EderSantana thoughts?"}}