{"action":"created","comment":{"url":"https://api.github.com/repos/Lasagne/Lasagne/pulls/comments/35100490","id":35100490,"diff_hunk":"@@ -0,0 +1,170 @@\n+'''\n+Recurrent network example.  Trains a bidirectional vanilla RNN to output the\n+sum of two numbers in a sequence of random numbers sampled uniformly from\n+[0, 1] based on a separate marker sequence.\n+'''\n+\n+from __future__ import print_function\n+\n+\n+import numpy as np\n+import theano\n+import theano.tensor as T\n+import lasagne\n+\n+\n+# Min/max sequence length\n+MIN_LENGTH = 50\n+MAX_LENGTH = 55\n+# Number of units in the hidden (recurrent) layer\n+N_HIDDEN = 100\n+# Number of training sequences in each batch\n+N_BATCH = 100\n+# Optimization learning rate\n+LEARNING_RATE = .001\n+# All gradients above this will be clipped\n+GRAD_CLIP = 100\n+# How often should we check the output?\n+EPOCH_SIZE = 100\n+# Number of epochs to train the net\n+NUM_EPOCHS = 10\n+\n+\n+def gen_data(min_length=MIN_LENGTH, max_length=MAX_LENGTH, n_batch=N_BATCH):\n+    '''\n+    Generate a batch of sequences for the \"add\" task, e.g. the target for the\n+    following\n+\n+    ``| 0.5 | 0.7 | 0.3 | 0.1 | 0.2 | ... | 0.5 | 0.9 | ... | 0.8 | 0.2 |\n+      |  0  |  0  |  1  |  0  |  0  |     |  0  |  1  |     |  0  |  0  |``\n+\n+    would be 0.3 + .9 = 1.2.  This task was proposed in [1]_ and explored in\n+    e.g. [2]_.\n+\n+    Parameters\n+    ----------\n+    min_length : int\n+        Minimum sequence length.\n+    max_length : int\n+        Maximum sequence length.\n+    n_batch : int\n+        Number of samples in the batch.\n+\n+    Returns\n+    -------\n+    X : np.ndarray\n+        Input to the network, of shape (n_batch, max_length, 2), where the last\n+        dimension corresponds to the two sequences shown above.\n+    y : np.ndarray\n+        Correct output for each sample, shape (n_batch,).\n+    mask : np.ndarray\n+        A binary matrix of shape (n_batch, max_length) where ``mask[i, j] = 1``\n+        when ``j <= (length of sequence i)`` and ``mask[i, j] = 0`` when ``j >\n+        (length of sequence i)``.","path":"examples/recurrent.py","position":63,"original_position":63,"commit_id":"3df009b5d05b809a23a55ad902a6a916d8480f31","original_commit_id":"3df009b5d05b809a23a55ad902a6a916d8480f31","user":{"login":"f0k","id":629706,"avatar_url":"https://avatars.githubusercontent.com/u/629706?v=3","gravatar_id":"","url":"https://api.github.com/users/f0k","html_url":"https://github.com/f0k","followers_url":"https://api.github.com/users/f0k/followers","following_url":"https://api.github.com/users/f0k/following{/other_user}","gists_url":"https://api.github.com/users/f0k/gists{/gist_id}","starred_url":"https://api.github.com/users/f0k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f0k/subscriptions","organizations_url":"https://api.github.com/users/f0k/orgs","repos_url":"https://api.github.com/users/f0k/repos","events_url":"https://api.github.com/users/f0k/events{/privacy}","received_events_url":"https://api.github.com/users/f0k/received_events","type":"User","site_admin":false},"body":"General note: I like the documentation of this function, it's very clear!","created_at":"2015-07-21T13:23:47Z","updated_at":"2015-07-21T13:23:47Z","html_url":"https://github.com/Lasagne/Lasagne/pull/294#discussion_r35100490","pull_request_url":"https://api.github.com/repos/Lasagne/Lasagne/pulls/294","_links":{"self":{"href":"https://api.github.com/repos/Lasagne/Lasagne/pulls/comments/35100490"},"html":{"href":"https://github.com/Lasagne/Lasagne/pull/294#discussion_r35100490"},"pull_request":{"href":"https://api.github.com/repos/Lasagne/Lasagne/pulls/294"}}},"pull_request":{"url":"https://api.github.com/repos/Lasagne/Lasagne/pulls/294","id":37486916,"html_url":"https://github.com/Lasagne/Lasagne/pull/294","diff_url":"https://github.com/Lasagne/Lasagne/pull/294.diff","patch_url":"https://github.com/Lasagne/Lasagne/pull/294.patch","issue_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/294","number":294,"state":"open","locked":false,"title":"Add recurrent layers","user":{"login":"craffel","id":417568,"avatar_url":"https://avatars.githubusercontent.com/u/417568?v=3","gravatar_id":"","url":"https://api.github.com/users/craffel","html_url":"https://github.com/craffel","followers_url":"https://api.github.com/users/craffel/followers","following_url":"https://api.github.com/users/craffel/following{/other_user}","gists_url":"https://api.github.com/users/craffel/gists{/gist_id}","starred_url":"https://api.github.com/users/craffel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/craffel/subscriptions","organizations_url":"https://api.github.com/users/craffel/orgs","repos_url":"https://api.github.com/users/craffel/repos","events_url":"https://api.github.com/users/craffel/events{/privacy}","received_events_url":"https://api.github.com/users/craffel/received_events","type":"User","site_admin":false},"body":"Addresses #17.  Includes layers for LSTM, GRU, \"vanilla\" (dense) RNN, and an RNN with arbitrary connection patterns.  Thanks to @skaae for the tests and the GRU layer (+more), and @JackKelly for some comment fixes and additional functionality.  Some misc comments that may warrant addressing -\r\n\r\n* We added examples to the examples directory.  Do you want these?  The usage of the recurrent layers is different enough that an example is probably warranted, but there is also an example in the top docstring of the recurrent submodule, which covers the absolute basics in terms of how to reshape things to make it work.  Any examples should probably be supplemented with more fleshed-out stuff, like @skaae's penn treebank example and my speech rec example.\r\n* We didn't resolve these two issues: https://github.com/craffel/nntools/issues/30 https://github.com/craffel/nntools/issues/15 which essentially are the same thing - i.e., the ability to use the RNNs for sequence prediction.  It's probably a feature we want to have, as it's a very hot topic at the moment.\r\n* The LSTM layer (and the GRU layer, to some extent) have a LOT of `__init__` arguments because there are so many things to initialize.  Most of the time people initialize all weight matrices in the same way, although I'm sure that's not always true.  It's actually quite common to initialize the forget gate bias vector differently than the rest.\r\n* The `peepholes` kwarg of the LSTM layer decides whether to include connections from the cell to the gates; sometimes people use these, sometimes they don't, so it's important that we include this functionality.  However, people also sometimes don't include other connections, and we don't have functionality for this.  What I'd really like is something where if one of the arguments is passed as `None`, then the connection is not used, but this would make the code pretty ugly - we'd essentially have to define a new dot method which does nothing when one of the arguments is `None` or something. So, in short, I'd rather not address this now.\r\n* The `grad_clipping` argument in all layers clips the gradients, which is very common for RNNs.  I think, however, this is pretty common for other network architectures too, so what I'd really like to see is it be moved out to a global function in updates, or something.  But, this isn't something I'm familiar with enough to know how feasible that is, @skaae would be better to comment on that.  \r\n* `hid_init`  (and `cell_init`) are allowed to be `TensorVariable`s, see our discussion here.  https://github.com/craffel/nntools/pull/27#issuecomment-107989355  To summarize that, I'd rather have them be able to be variable-size shared variables which are added with `add_param`, but we can't because of the way we allow for variable batch size.  Any ideas on that discussion are appreciated.\r\n\r\nOk, that's about it, let me know what you think!  Would be great to get this merged before the first release.","created_at":"2015-06-11T17:15:40Z","updated_at":"2015-07-21T13:23:47Z","closed_at":null,"merged_at":null,"merge_commit_sha":"c74ddd25e78c4a9a52499c6d5e880e8f13a26440","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/Lasagne/Lasagne/pulls/294/commits","review_comments_url":"https://api.github.com/repos/Lasagne/Lasagne/pulls/294/comments","review_comment_url":"https://api.github.com/repos/Lasagne/Lasagne/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/294/comments","statuses_url":"https://api.github.com/repos/Lasagne/Lasagne/statuses/3df009b5d05b809a23a55ad902a6a916d8480f31","head":{"label":"craffel:recurrent","ref":"recurrent","sha":"3df009b5d05b809a23a55ad902a6a916d8480f31","user":{"login":"craffel","id":417568,"avatar_url":"https://avatars.githubusercontent.com/u/417568?v=3","gravatar_id":"","url":"https://api.github.com/users/craffel","html_url":"https://github.com/craffel","followers_url":"https://api.github.com/users/craffel/followers","following_url":"https://api.github.com/users/craffel/following{/other_user}","gists_url":"https://api.github.com/users/craffel/gists{/gist_id}","starred_url":"https://api.github.com/users/craffel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/craffel/subscriptions","organizations_url":"https://api.github.com/users/craffel/orgs","repos_url":"https://api.github.com/users/craffel/repos","events_url":"https://api.github.com/users/craffel/events{/privacy}","received_events_url":"https://api.github.com/users/craffel/received_events","type":"User","site_admin":false},"repo":{"id":24106437,"name":"nntools","full_name":"craffel/nntools","owner":{"login":"craffel","id":417568,"avatar_url":"https://avatars.githubusercontent.com/u/417568?v=3","gravatar_id":"","url":"https://api.github.com/users/craffel","html_url":"https://github.com/craffel","followers_url":"https://api.github.com/users/craffel/followers","following_url":"https://api.github.com/users/craffel/following{/other_user}","gists_url":"https://api.github.com/users/craffel/gists{/gist_id}","starred_url":"https://api.github.com/users/craffel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/craffel/subscriptions","organizations_url":"https://api.github.com/users/craffel/orgs","repos_url":"https://api.github.com/users/craffel/repos","events_url":"https://api.github.com/users/craffel/events{/privacy}","received_events_url":"https://api.github.com/users/craffel/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/craffel/nntools","description":"Adding recurrent layers to lasagne","fork":true,"url":"https://api.github.com/repos/craffel/nntools","forks_url":"https://api.github.com/repos/craffel/nntools/forks","keys_url":"https://api.github.com/repos/craffel/nntools/keys{/key_id}","collaborators_url":"https://api.github.com/repos/craffel/nntools/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/craffel/nntools/teams","hooks_url":"https://api.github.com/repos/craffel/nntools/hooks","issue_events_url":"https://api.github.com/repos/craffel/nntools/issues/events{/number}","events_url":"https://api.github.com/repos/craffel/nntools/events","assignees_url":"https://api.github.com/repos/craffel/nntools/assignees{/user}","branches_url":"https://api.github.com/repos/craffel/nntools/branches{/branch}","tags_url":"https://api.github.com/repos/craffel/nntools/tags","blobs_url":"https://api.github.com/repos/craffel/nntools/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/craffel/nntools/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/craffel/nntools/git/refs{/sha}","trees_url":"https://api.github.com/repos/craffel/nntools/git/trees{/sha}","statuses_url":"https://api.github.com/repos/craffel/nntools/statuses/{sha}","languages_url":"https://api.github.com/repos/craffel/nntools/languages","stargazers_url":"https://api.github.com/repos/craffel/nntools/stargazers","contributors_url":"https://api.github.com/repos/craffel/nntools/contributors","subscribers_url":"https://api.github.com/repos/craffel/nntools/subscribers","subscription_url":"https://api.github.com/repos/craffel/nntools/subscription","commits_url":"https://api.github.com/repos/craffel/nntools/commits{/sha}","git_commits_url":"https://api.github.com/repos/craffel/nntools/git/commits{/sha}","comments_url":"https://api.github.com/repos/craffel/nntools/comments{/number}","issue_comment_url":"https://api.github.com/repos/craffel/nntools/issues/comments{/number}","contents_url":"https://api.github.com/repos/craffel/nntools/contents/{+path}","compare_url":"https://api.github.com/repos/craffel/nntools/compare/{base}...{head}","merges_url":"https://api.github.com/repos/craffel/nntools/merges","archive_url":"https://api.github.com/repos/craffel/nntools/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/craffel/nntools/downloads","issues_url":"https://api.github.com/repos/craffel/nntools/issues{/number}","pulls_url":"https://api.github.com/repos/craffel/nntools/pulls{/number}","milestones_url":"https://api.github.com/repos/craffel/nntools/milestones{/number}","notifications_url":"https://api.github.com/repos/craffel/nntools/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/craffel/nntools/labels{/name}","releases_url":"https://api.github.com/repos/craffel/nntools/releases{/id}","created_at":"2014-09-16T15:54:57Z","updated_at":"2015-07-18T20:52:23Z","pushed_at":"2015-07-19T17:15:22Z","git_url":"git://github.com/craffel/nntools.git","ssh_url":"git@github.com:craffel/nntools.git","clone_url":"https://github.com/craffel/nntools.git","svn_url":"https://github.com/craffel/nntools","homepage":"","size":3612,"stargazers_count":53,"watchers_count":53,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":10,"mirror_url":null,"open_issues_count":5,"forks":10,"open_issues":5,"watchers":53,"default_branch":"master"}},"base":{"label":"Lasagne:master","ref":"master","sha":"bb70e48f44ed7e907ee3986d60bc1ba9183e4263","user":{"login":"Lasagne","id":11752110,"avatar_url":"https://avatars.githubusercontent.com/u/11752110?v=3","gravatar_id":"","url":"https://api.github.com/users/Lasagne","html_url":"https://github.com/Lasagne","followers_url":"https://api.github.com/users/Lasagne/followers","following_url":"https://api.github.com/users/Lasagne/following{/other_user}","gists_url":"https://api.github.com/users/Lasagne/gists{/gist_id}","starred_url":"https://api.github.com/users/Lasagne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Lasagne/subscriptions","organizations_url":"https://api.github.com/users/Lasagne/orgs","repos_url":"https://api.github.com/users/Lasagne/repos","events_url":"https://api.github.com/users/Lasagne/events{/privacy}","received_events_url":"https://api.github.com/users/Lasagne/received_events","type":"Organization","site_admin":false},"repo":{"id":23924195,"name":"Lasagne","full_name":"Lasagne/Lasagne","owner":{"login":"Lasagne","id":11752110,"avatar_url":"https://avatars.githubusercontent.com/u/11752110?v=3","gravatar_id":"","url":"https://api.github.com/users/Lasagne","html_url":"https://github.com/Lasagne","followers_url":"https://api.github.com/users/Lasagne/followers","following_url":"https://api.github.com/users/Lasagne/following{/other_user}","gists_url":"https://api.github.com/users/Lasagne/gists{/gist_id}","starred_url":"https://api.github.com/users/Lasagne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Lasagne/subscriptions","organizations_url":"https://api.github.com/users/Lasagne/orgs","repos_url":"https://api.github.com/users/Lasagne/repos","events_url":"https://api.github.com/users/Lasagne/events{/privacy}","received_events_url":"https://api.github.com/users/Lasagne/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Lasagne/Lasagne","description":"neural network tools for Theano","fork":false,"url":"https://api.github.com/repos/Lasagne/Lasagne","forks_url":"https://api.github.com/repos/Lasagne/Lasagne/forks","keys_url":"https://api.github.com/repos/Lasagne/Lasagne/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Lasagne/Lasagne/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Lasagne/Lasagne/teams","hooks_url":"https://api.github.com/repos/Lasagne/Lasagne/hooks","issue_events_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/events{/number}","events_url":"https://api.github.com/repos/Lasagne/Lasagne/events","assignees_url":"https://api.github.com/repos/Lasagne/Lasagne/assignees{/user}","branches_url":"https://api.github.com/repos/Lasagne/Lasagne/branches{/branch}","tags_url":"https://api.github.com/repos/Lasagne/Lasagne/tags","blobs_url":"https://api.github.com/repos/Lasagne/Lasagne/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Lasagne/Lasagne/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Lasagne/Lasagne/git/refs{/sha}","trees_url":"https://api.github.com/repos/Lasagne/Lasagne/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Lasagne/Lasagne/statuses/{sha}","languages_url":"https://api.github.com/repos/Lasagne/Lasagne/languages","stargazers_url":"https://api.github.com/repos/Lasagne/Lasagne/stargazers","contributors_url":"https://api.github.com/repos/Lasagne/Lasagne/contributors","subscribers_url":"https://api.github.com/repos/Lasagne/Lasagne/subscribers","subscription_url":"https://api.github.com/repos/Lasagne/Lasagne/subscription","commits_url":"https://api.github.com/repos/Lasagne/Lasagne/commits{/sha}","git_commits_url":"https://api.github.com/repos/Lasagne/Lasagne/git/commits{/sha}","comments_url":"https://api.github.com/repos/Lasagne/Lasagne/comments{/number}","issue_comment_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/comments{/number}","contents_url":"https://api.github.com/repos/Lasagne/Lasagne/contents/{+path}","compare_url":"https://api.github.com/repos/Lasagne/Lasagne/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Lasagne/Lasagne/merges","archive_url":"https://api.github.com/repos/Lasagne/Lasagne/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Lasagne/Lasagne/downloads","issues_url":"https://api.github.com/repos/Lasagne/Lasagne/issues{/number}","pulls_url":"https://api.github.com/repos/Lasagne/Lasagne/pulls{/number}","milestones_url":"https://api.github.com/repos/Lasagne/Lasagne/milestones{/number}","notifications_url":"https://api.github.com/repos/Lasagne/Lasagne/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Lasagne/Lasagne/labels{/name}","releases_url":"https://api.github.com/repos/Lasagne/Lasagne/releases{/id}","created_at":"2014-09-11T15:31:41Z","updated_at":"2015-07-20T18:07:23Z","pushed_at":"2015-07-20T16:43:56Z","git_url":"git://github.com/Lasagne/Lasagne.git","ssh_url":"git@github.com:Lasagne/Lasagne.git","clone_url":"https://github.com/Lasagne/Lasagne.git","svn_url":"https://github.com/Lasagne/Lasagne","homepage":"http://lasagne.readthedocs.org/","size":3755,"stargazers_count":777,"watchers_count":777,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":184,"mirror_url":null,"open_issues_count":37,"forks":184,"open_issues":37,"watchers":777,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/Lasagne/Lasagne/pulls/294"},"html":{"href":"https://github.com/Lasagne/Lasagne/pull/294"},"issue":{"href":"https://api.github.com/repos/Lasagne/Lasagne/issues/294"},"comments":{"href":"https://api.github.com/repos/Lasagne/Lasagne/issues/294/comments"},"review_comments":{"href":"https://api.github.com/repos/Lasagne/Lasagne/pulls/294/comments"},"review_comment":{"href":"https://api.github.com/repos/Lasagne/Lasagne/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Lasagne/Lasagne/pulls/294/commits"},"statuses":{"href":"https://api.github.com/repos/Lasagne/Lasagne/statuses/3df009b5d05b809a23a55ad902a6a916d8480f31"}}}}