{"action":"created","issue":{"url":"https://api.github.com/repos/mila-udem/blocks-examples/issues/20","labels_url":"https://api.github.com/repos/mila-udem/blocks-examples/issues/20/labels{/name}","comments_url":"https://api.github.com/repos/mila-udem/blocks-examples/issues/20/comments","events_url":"https://api.github.com/repos/mila-udem/blocks-examples/issues/20/events","html_url":"https://github.com/mila-udem/blocks-examples/pull/20","id":92208176,"number":20,"title":"NMT","user":{"login":"orhanf","id":3654684,"avatar_url":"https://avatars.githubusercontent.com/u/3654684?v=3","gravatar_id":"","url":"https://api.github.com/users/orhanf","html_url":"https://github.com/orhanf","followers_url":"https://api.github.com/users/orhanf/followers","following_url":"https://api.github.com/users/orhanf/following{/other_user}","gists_url":"https://api.github.com/users/orhanf/gists{/gist_id}","starred_url":"https://api.github.com/users/orhanf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/orhanf/subscriptions","organizations_url":"https://api.github.com/users/orhanf/orgs","repos_url":"https://api.github.com/users/orhanf/repos","events_url":"https://api.github.com/users/orhanf/events{/privacy}","received_events_url":"https://api.github.com/users/orhanf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-06-30T21:43:12Z","updated_at":"2015-07-09T11:48:57Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/mila-udem/blocks-examples/pulls/20","html_url":"https://github.com/mila-udem/blocks-examples/pull/20","diff_url":"https://github.com/mila-udem/blocks-examples/pull/20.diff","patch_url":"https://github.com/mila-udem/blocks-examples/pull/20.patch"},"body":"@dmitriy-serdyuk, this is the initial implementation of RNN encoder-decoder with attention for machine translation. Working with the following versions (latest commits June30, 2015): [blocks](https://github.com/mila-udem/blocks/tree/39184e2c559a3a469f75399ddd4b8b6557bccdb8), [fuel](https://github.com/mila-udem/fuel/tree/ba15181b357035ee19ae32953f0c7c77f4ad490d) and [picklable_itertools](https://github.com/mila-udem/picklable-itertools/tree/5de7171788524eb6521117652e5fa09b45f74023)\r\n\r\nItems TODO:\r\n- [ ] Documentation; is not complete yet but decent\r\n- [ ] Sampling; not tested but should be fine\r\n- [ ] Early stopping based on BLEU; not tested, have to clean it up and adapt the changes/fixes from NMT\r\n- [ ] Example is using WMT'15 Czech->English translation, but necessary input files (preprocessed bitext, vocabularies, validation sets) are not provided. This will be handled by adding a script that does all the pre-processing and puts everything to the corresponding folder. Have the scripts that we used for WMT'15, will clean it up and put it here soon.\r\n- [ ] Anything else ?\r\n\r\nI will continue on these items this week, all minor issues compared to the whole PR\r\n\r\n@ejls, can you please take a look if i am missing something.\r\n@rizar your comments/recommendations are also highly welcomed \r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/mila-udem/blocks-examples/issues/comments/119924863","html_url":"https://github.com/mila-udem/blocks-examples/pull/20#issuecomment-119924863","issue_url":"https://api.github.com/repos/mila-udem/blocks-examples/issues/20","id":119924863,"user":{"login":"fhirschmann","id":12200,"avatar_url":"https://avatars.githubusercontent.com/u/12200?v=3","gravatar_id":"","url":"https://api.github.com/users/fhirschmann","html_url":"https://github.com/fhirschmann","followers_url":"https://api.github.com/users/fhirschmann/followers","following_url":"https://api.github.com/users/fhirschmann/following{/other_user}","gists_url":"https://api.github.com/users/fhirschmann/gists{/gist_id}","starred_url":"https://api.github.com/users/fhirschmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fhirschmann/subscriptions","organizations_url":"https://api.github.com/users/fhirschmann/orgs","repos_url":"https://api.github.com/users/fhirschmann/repos","events_url":"https://api.github.com/users/fhirschmann/events{/privacy}","received_events_url":"https://api.github.com/users/fhirschmann/received_events","type":"User","site_admin":false},"created_at":"2015-07-09T11:48:57Z","updated_at":"2015-07-09T11:48:57Z","body":"Another issue I found that may only be limited to sampling, but is more likely an issue for the NMT model itself:\r\n\r\nIn [stream_cs2en.py:51](https://github.com/orhanf/blocks-examples/blob/nmt/machine_translation/stream_cs2en.py#L51) you set the end-of-sequence marker to the size of the vocabulary. However, the EOS marker is never actually added when the model is computed. In GroundHog this was solved by [setting the last element in the sequence](https://github.com/lisa-groundhog/GroundHog/blob/master/experiments/nmt/encdec.py#L1509) to the EOS token, and indeed there are some remains in [sampling.py:42](https://github.com/lisa-groundhog/GroundHog/blob/master/experiments/nmt/encdec.py#L1509) which do not seem to get executed at all.\r\n\r\nAn example input sequence now looks like this (with a vocabulary size of 220):\r\n```\r\narray([[ 22, 114,  11,  23, 143,   2,  10, 156,  89,   1,  27,  32,  33,\r\n         38, 165, 119,   2, 137,  85, 154,  63, 120,  54, 208,   6, 182,\r\n          2,  20,   8,  83,   1,   1,   3,   1,   0,   0,   0,   0,   0,\r\n          0,   0,   0,   0]])\r\n\r\n```\r\n\r\nLikewise, a sequence does not start with a BOS token, but I believe this was also the case in GroundHog. I also noticed that, disregarding the 0-padding, all sequences end with 1 (the UNK token)."}}