{"action":"created","issue":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/425","labels_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/425/labels{/name}","comments_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/425/comments","events_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/425/events","html_url":"https://github.com/Lasagne/Lasagne/issues/425","id":107537894,"number":425,"title":"Provide RNN \"containers\"","user":{"login":"f0k","id":629706,"avatar_url":"https://avatars.githubusercontent.com/u/629706?v=3","gravatar_id":"","url":"https://api.github.com/users/f0k","html_url":"https://github.com/f0k","followers_url":"https://api.github.com/users/f0k/followers","following_url":"https://api.github.com/users/f0k/following{/other_user}","gists_url":"https://api.github.com/users/f0k/gists{/gist_id}","starred_url":"https://api.github.com/users/f0k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f0k/subscriptions","organizations_url":"https://api.github.com/users/f0k/orgs","repos_url":"https://api.github.com/users/f0k/repos","events_url":"https://api.github.com/users/f0k/events{/privacy}","received_events_url":"https://api.github.com/users/f0k/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":40,"created_at":"2015-09-21T15:31:14Z","updated_at":"2015-11-18T13:13:44Z","closed_at":null,"body":"In https://github.com/Lasagne/Recipes/pull/23#issuecomment-141979978, @skaae brought up the idea of splitting up the recurrent layers into layers performing a single computation step and a container layer that handles the recurrence (i.e., sets up the `scan` function or explicitly unrolls it).\r\n\r\nThe use case was to provide a way to efficiently sample from an RNN that was trained to generate time sequences (e.g., text) -- the most efficient way would be to tap into the scan function, but that's impossible the way it's currently encapsulated.\r\nAnother advantage could be that it reduces the amount of code duplication (see #356), and maybe it would even make it easier to equip LSTMs or GRUs with custom connections (#357)?\r\n\r\nAs far as I've understood, the following current code:\r\n```\r\nlayer = lasagne.layers.InputLayer(shape=(N_BATCH, MAX_LENGTH, 2))\r\nlayer = lasagne.layers.RecurrentLayer(layer, N_HIDDEN)\r\nlayer = lasagne.layers.SliceLayer(layer, -1, 1)\r\nlayer = lasagne.layers.DenseLayer(layer, num_units=1, nonlinearity=lasagne.nonlinearities.tanh)\r\n```\r\nCould be turned into:\r\n```\r\nlayer = lasagne.layers.InputLayer(shape=(N_BATCH, MAX_LENGTH, 2))\r\n# single time step\r\nrnn_layer = lasagne.layers.InputLayer((N_BATCH, 2))\r\nrnn_layer = lasagne.layers.VanillaRecurrentStep(rnn_layer, N_HIDDEN)\r\nlayer = lasagne.layers.Recurrence(layer, step=rnn_layer)\r\nlayer = lasagne.layers.SliceLayer(layer, -1, 1)\r\nlayer = lasagne.layers.DenseLayer(layer, num_units=1, nonlinearity=lasagne.nonlinearities.tanh)\r\n```\r\nWhere we'd have different `Step` layers, and one of them could be sampling from a multivariate distribution, for the character generation use case.\r\n`Recurrence` could have a keyword argument telling whether it persists its recurrent states, and another telling whether it feeds back on its own output (that would be needed for character generation).\r\n\r\nDisclaimer: I've just started looking into RNNs and am not sure what's the best way to break down the code yet -- I'm just opening this issue so there is a place to discuss the idea for those who are into RNNs with possible participation by a future me :)"},"comment":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/comments/157708958","html_url":"https://github.com/Lasagne/Lasagne/issues/425#issuecomment-157708958","issue_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/425","id":157708958,"user":{"login":"f0k","id":629706,"avatar_url":"https://avatars.githubusercontent.com/u/629706?v=3","gravatar_id":"","url":"https://api.github.com/users/f0k","html_url":"https://github.com/f0k","followers_url":"https://api.github.com/users/f0k/followers","following_url":"https://api.github.com/users/f0k/following{/other_user}","gists_url":"https://api.github.com/users/f0k/gists{/gist_id}","starred_url":"https://api.github.com/users/f0k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f0k/subscriptions","organizations_url":"https://api.github.com/users/f0k/orgs","repos_url":"https://api.github.com/users/f0k/repos","events_url":"https://api.github.com/users/f0k/events{/privacy}","received_events_url":"https://api.github.com/users/f0k/received_events","type":"User","site_admin":false},"created_at":"2015-11-18T13:13:44Z","updated_at":"2015-11-18T13:13:44Z","body":"> My intention would be that RecurrentStepLayer.get_output_for would return a list of sequences, and then leave it up to the user to index out the outputs they want before feeding it into the rest of the network.\r\n\r\nIf the `RecurrentStepLayer` is a layer directly in the DAG (i.e., included in `get_all_layers(output_layer)`), then we need to come up with a solution for #337 and adapt `get_output()` accordingly. If the `RecurrentStepLayer` is hidden from `get_all_layers()`, then we can do what we want, but it would still be better to decide on #337 first to have a unified API.\r\n\r\n> At any rate, the point is that if a user is writing \"user code\" with a `RecurrentStepLayer` (or whatever the object is which calls scan given a step function), I think it's reasonable to ask them to do some custom behavior which is not just stacking layers.\r\n\r\nSure, I'm not opposed to that. But whatever we implement for `RecurrentStepLayer` to return multiple outputs to choose from must be supported indefinitely as soon as we include it in a released version of Lasagne. If we implement it to return a list of outputs and later decide in #337 that the best solution for multi-output layers would be to return a dictionary of outputs, we'd end up having to support both ways for `RecurrentStepLayer`. That's why I think we should finish discussing #337 before merging RNN containers into Lasagne -- it's a pretty minor point wrt. the RNN container API, though, we can still think it through now!\r\n\r\n> PS: Here managed to shrink the whole recurrent.py to about 400 lines.\r\n\r\nNice, but to replace `recurrent.py`, it will still need to support the original API (i.e., include a CustomRecurrentLayer etc.) and pass the original tests. It's a good basis for discussing RNN containers, though, thanks a lot!\r\n\r\n> Again more suggestions and advices are welcome.\r\n\r\nJust read the `RecurrentStepLayer`. It's a good start, but it still has some drawbacks.\r\nIt assumes that the `StepLayer` has a `W_in` and `b_in` attribute for preprocessing the input. This is something the vanilla, LSTM and GRU recurrent layers have in common (so it's perfect for reconciling those three), but it leaves out the `CustomRecurrentLayer`. For example, it makes it impossible to implement a Convolutional RNN, or a Convolutional LSTM.\r\nFurthermore, I think one idea for the `RecurrentStepLayer` would be to allow it to combine a deep RNN into a single scan, i.e., allow it to process a stack of LSTMs at once.\r\n\r\nTo be really general, I guess the `step_layer` supplied to the `RecurrentStepLayer` must be a network that accepts multiple inputs (at least one of them would be recurrent, and at most one of them would be non-recurrent) and produces multiple outputs (namely those that are recurrent). This would allow many-to-one and one-to-many mappings, including those that recurrently feed back some processed output (such as the previously predicted word). Intuitively, I'd say that the initialization of hidden states would be a job for the `RecurrentStepLayer` then, so `step_layer` can really be an arbitrary DAG of Lasagne layers, not a special-purpose class as in your current code.\r\nThis would mean that the vanilla step layer, LSTM step layer and GRU step layer would have to be ordinary layers, or conglomerates of layers.\r\n\r\nDoes that sound like a useful goal? What would we need to figure out for that to work?"}}