{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/957","labels_url":"https://api.github.com/repos/fchollet/keras/issues/957/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/957/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/957/events","html_url":"https://github.com/fchollet/keras/issues/957","id":115411200,"number":957,"title":"Using Masking Layer for Sequence to Sequence Learning","user":{"login":"LeavesBreathe","id":14891677,"avatar_url":"https://avatars.githubusercontent.com/u/14891677?v=3","gravatar_id":"","url":"https://api.github.com/users/LeavesBreathe","html_url":"https://github.com/LeavesBreathe","followers_url":"https://api.github.com/users/LeavesBreathe/followers","following_url":"https://api.github.com/users/LeavesBreathe/following{/other_user}","gists_url":"https://api.github.com/users/LeavesBreathe/gists{/gist_id}","starred_url":"https://api.github.com/users/LeavesBreathe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LeavesBreathe/subscriptions","organizations_url":"https://api.github.com/users/LeavesBreathe/orgs","repos_url":"https://api.github.com/users/LeavesBreathe/repos","events_url":"https://api.github.com/users/LeavesBreathe/events{/privacy}","received_events_url":"https://api.github.com/users/LeavesBreathe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-11-06T00:57:01Z","updated_at":"2015-11-06T14:56:08Z","closed_at":null,"body":"Hey Guys,\r\n\r\nI know @EderSantana asked this question a few months ago https://github.com/fchollet/keras/issues/395, and I have read several Keras threads that deal with masking.\r\n\r\nThe question was: We can mask the input of zeros, but won't it bias the cost function as we are not masking the zeros on the *output*?\r\n\r\nLater, the ```Masking``` layer was added, and I do believe that this output problem was resolved. However, in order to use masking *without* an embedding layer, my model structure is:\r\n\r\n\tmodel = Sequential()\r\n\tM = Masking(mask_value=0.)\r\n\tM._input_shape = (x_maxlen, word2vec_dimension)\r\n\tmodel.add(M)\r\n    model.add(LSTM(hidden_variables_encoding, return_sequences = False)) \r\n    model.add(Dense(hidden_variables_encoding)) \r\n    model.add(Activation('relu'))\r\n    model.add(RepeatVector(y_maxlen))\r\n    for z in range(0,number_of_decoding_layers):\r\n        model.add(LSTM(hidden_variables_decoding, return_sequences=True))\r\n        model.add(Dropout(dropout))\r\n    model.add(TimeDistributedDense(y_matrix_axis, activation=\"softmax\")) \r\n    model.compile(loss='categorical_crossentropy', optimizer='adam')\r\n\r\n\r\nI have two questions:\r\n\r\n* My input is in vector form, so I have '0.' (notice the float) for masking my input. My output, however is in integer numpy matrix. So, does the '0.' cover the number '0' in an integer numpy matrix? Or maybe I should use ```mask_value = 0``` (no floating)\r\n\r\n* Given the model I've presented, does the masking layer mask the zeros that come out of the ```TimeDistributedDense``` layer? If not, this would lead to the cost function being biased by zeros!\r\n\r\nI apologize for the redundancy of the issue, but it is a very important one. Thank you!"},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/154429995","html_url":"https://github.com/fchollet/keras/issues/957#issuecomment-154429995","issue_url":"https://api.github.com/repos/fchollet/keras/issues/957","id":154429995,"user":{"login":"LeavesBreathe","id":14891677,"avatar_url":"https://avatars.githubusercontent.com/u/14891677?v=3","gravatar_id":"","url":"https://api.github.com/users/LeavesBreathe","html_url":"https://github.com/LeavesBreathe","followers_url":"https://api.github.com/users/LeavesBreathe/followers","following_url":"https://api.github.com/users/LeavesBreathe/following{/other_user}","gists_url":"https://api.github.com/users/LeavesBreathe/gists{/gist_id}","starred_url":"https://api.github.com/users/LeavesBreathe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LeavesBreathe/subscriptions","organizations_url":"https://api.github.com/users/LeavesBreathe/orgs","repos_url":"https://api.github.com/users/LeavesBreathe/repos","events_url":"https://api.github.com/users/LeavesBreathe/events{/privacy}","received_events_url":"https://api.github.com/users/LeavesBreathe/received_events","type":"User","site_admin":false},"created_at":"2015-11-06T14:56:08Z","updated_at":"2015-11-06T14:56:08Z","body":">I'm preparing a tutorial. I need you guys to understand sequence to sequence learning to be able to use a new recurrent layer that is coming up... so much mystery... but I believe you will like it.\r\n\r\nNever been more excited for a tutorial in my life.\r\n\r\n>I'm writing my thesis proposal and I got a carpal tunnel... \r\n\r\nI actually went to med school for a while, so if you ever want to chat, I wouldn't mind giving you some helpful suggestions for carpal tunnel. I actually know a ton about it as I started to get it a while back. \r\n"}}