{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/395","labels_url":"https://api.github.com/repos/fchollet/keras/issues/395/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/395/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/395/events","html_url":"https://github.com/fchollet/keras/issues/395","id":95211755,"number":395,"title":"is the Sequence to Sequence learning right?","user":{"login":"EderSantana","id":3902480,"avatar_url":"https://avatars.githubusercontent.com/u/3902480?v=3","gravatar_id":"","url":"https://api.github.com/users/EderSantana","html_url":"https://github.com/EderSantana","followers_url":"https://api.github.com/users/EderSantana/followers","following_url":"https://api.github.com/users/EderSantana/following{/other_user}","gists_url":"https://api.github.com/users/EderSantana/gists{/gist_id}","starred_url":"https://api.github.com/users/EderSantana/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/EderSantana/subscriptions","organizations_url":"https://api.github.com/users/EderSantana/orgs","repos_url":"https://api.github.com/users/EderSantana/repos","events_url":"https://api.github.com/users/EderSantana/events{/privacy}","received_events_url":"https://api.github.com/users/EderSantana/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":162,"created_at":"2015-07-15T14:38:31Z","updated_at":"2015-11-10T17:36:15Z","closed_at":"2015-08-13T14:33:52Z","body":"Assume we are trying to learn a sequence to sequence map. For this we can use Recurrent and TimeDistributedDense layers. Now assume that the sequences have different lengths. We should pad both input and desired sequences with zeros, right? But how will the objective function handle the  padded values? There is no choice to pass a mask to the objective function. Won't this bias the cost function?"},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/155502174","html_url":"https://github.com/fchollet/keras/issues/395#issuecomment-155502174","issue_url":"https://api.github.com/repos/fchollet/keras/issues/395","id":155502174,"user":{"login":"gautamb85","id":1865082,"avatar_url":"https://avatars.githubusercontent.com/u/1865082?v=3","gravatar_id":"","url":"https://api.github.com/users/gautamb85","html_url":"https://github.com/gautamb85","followers_url":"https://api.github.com/users/gautamb85/followers","following_url":"https://api.github.com/users/gautamb85/following{/other_user}","gists_url":"https://api.github.com/users/gautamb85/gists{/gist_id}","starred_url":"https://api.github.com/users/gautamb85/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gautamb85/subscriptions","organizations_url":"https://api.github.com/users/gautamb85/orgs","repos_url":"https://api.github.com/users/gautamb85/repos","events_url":"https://api.github.com/users/gautamb85/events{/privacy}","received_events_url":"https://api.github.com/users/gautamb85/received_events","type":"User","site_admin":false},"created_at":"2015-11-10T17:36:15Z","updated_at":"2015-11-10T17:36:15Z","body":"@farizrahman4u \r\nLets say I have an english-french translation pair. So you would encode the english sentence (input 1). \r\nFor the decoder, in Cho's paper (appendix) the GRU non-linearity gets 3 inputs\r\n\r\n - the context/encoding produced by the encoder (and its associated weight matrix)\r\n- The french sentence (which at test time is replaced by the prediction)\r\n- The hidden activation from the previous time-step.\r\n\r\n1. Providing an extra input (so that the context vector is fed explicitly to each time-step) might prove a little problematic as it might need a new layer to be written. \r\nYou can refer to https://github.com/dagbldr/dagbldr\r\nHe has a conditional_GRU layer Vs  a standard GRU_layer \r\n- Though a workaround might be to just use the encoding to initialize the decoder recurrence (like in sutskiver's paper)\r\n\r\n2. If one wants to use the model to generate translations, then the prediction is needed at test time. If the model is to be used for scoring, we just need to evaluate the log-likelihood / negative cost\r\n\r\n- optimization criteria etc are exactly the same.\r\n\r\n- I had an idea how to do this using a graph() model, but the problem was getting an additional input to a  GRU/LSTM layer. Plus I couldn't initialize the decoder recurrence as I didn't have a stateful RNN. So I can try that out. \r\n\r\nDoes a graph() model make sense for what I described? Il post some pseudo code in a little while)"}}