{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/36","labels_url":"https://api.github.com/repos/fchollet/keras/issues/36/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/36/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/36/events","html_url":"https://github.com/fchollet/keras/pull/36","id":66471631,"number":36,"title":"Advanced Reshape Layer","user":{"login":"patyork","id":1304633,"avatar_url":"https://avatars.githubusercontent.com/u/1304633?v=3","gravatar_id":"","url":"https://api.github.com/users/patyork","html_url":"https://github.com/patyork","followers_url":"https://api.github.com/users/patyork/followers","following_url":"https://api.github.com/users/patyork/following{/other_user}","gists_url":"https://api.github.com/users/patyork/gists{/gist_id}","starred_url":"https://api.github.com/users/patyork/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/patyork/subscriptions","organizations_url":"https://api.github.com/users/patyork/orgs","repos_url":"https://api.github.com/users/patyork/repos","events_url":"https://api.github.com/users/patyork/events{/privacy}","received_events_url":"https://api.github.com/users/patyork/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-04-05T19:12:00Z","updated_at":"2015-04-05T20:15:58Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/fchollet/keras/pulls/36","html_url":"https://github.com/fchollet/keras/pull/36","diff_url":"https://github.com/fchollet/keras/pull/36.diff","patch_url":"https://github.com/fchollet/keras/pull/36.patch"},"body":"I've added a bit of an architecture change, as well as a new layer to allow Advanced Reshaping.\r\n\r\nBasically, to allow reshapes involving the first dimension, the number of samples in the current batch (`current_batch_size`) must be available to the AdvancedReshape layer. The easiest way that I could see to allow that is to throw a new parameter (`current_batch_size`) to the `Layer.output` function. This recursively passes the current number of samples in the batch to each layer, so that it has it available if it is necessary. This required touching every layer.\r\n\r\nThe AdvancedReshape layer takes a lambda expression for the initialization parameter. This lambda should take 2 arguments: current_batch_size and current_shape (can be seen below). From these parameters, it is possible to reshape between 1D, 2D, 3D (and possibly to ND, although I am unsure on that). *This lambda should return a tuple.*\r\n\r\nBelow is an example in which the current_batch_size changes on the pass over the last batch. It is simplistic and not overly useful, but it shows that AdvancedReshape can correctly reshape from 2D -> 3D and then from 3D -> 2D without breaking over the toughest example:\r\n```python\r\nmodel = Sequential()\r\nmodel.add(Dense(2,1, activation='sigmoid'))\r\n\r\n# Reshape to 3D: (number of sample in current batch, elements in each sample, values)\r\nmodel.add(AdvancedReshape(new_shape_fn=lambda current_batch_size, current_shape: (current_batch_size, current_shape[0]/current_batch_size, current_shape[1])))\r\n# Reshape back to 2D ((number of sample in current batch * elements in each sample, values))\r\nmodel.add(AdvancedReshape(new_shape_fn=lambda current_batch_size, current_shape: (current_batch_size * current_shape[1], current_shape[2])))\r\n\r\nsgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)\r\nmodel.compile(loss='mse', optimizer='sgd')\r\n\r\nX = np.zeros((3,2))\r\nY = np.zeros((3,1))\r\nmodel.fit(X, Y, batch_size=2, nb_epoch=1)\r\n```\r\n```\r\nEpoch 0\r\n\b2/3 [===================>..........] - ETA: 0s - loss: 0.2500\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r\n3/3 [==============================] - 0s - loss: 0.2497\r\n```"},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/89846574","html_url":"https://github.com/fchollet/keras/pull/36#issuecomment-89846574","issue_url":"https://api.github.com/repos/fchollet/keras/issues/36","id":89846574,"user":{"login":"patyork","id":1304633,"avatar_url":"https://avatars.githubusercontent.com/u/1304633?v=3","gravatar_id":"","url":"https://api.github.com/users/patyork","html_url":"https://github.com/patyork","followers_url":"https://api.github.com/users/patyork/followers","following_url":"https://api.github.com/users/patyork/following{/other_user}","gists_url":"https://api.github.com/users/patyork/gists{/gist_id}","starred_url":"https://api.github.com/users/patyork/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/patyork/subscriptions","organizations_url":"https://api.github.com/users/patyork/orgs","repos_url":"https://api.github.com/users/patyork/repos","events_url":"https://api.github.com/users/patyork/events{/privacy}","received_events_url":"https://api.github.com/users/patyork/received_events","type":"User","site_admin":false},"created_at":"2015-04-05T20:15:58Z","updated_at":"2015-04-05T20:15:58Z","body":"A less trivial example (and, in fact, one that is quite useful). The below is a deep net, with 3 Dense layers, a recurrent layer, and a Dense layer:\r\n```python\r\n# This model is similar to the architecture utilized in DeepSpeech [http://arxiv.org/abs/1412.5567]\r\n#   which attained state-of-the art performance in speech recognition in noisy environments\r\n# The only changes would be:\r\n#   -a Bidirectional RNN (BRNN) instead of a simple RNN layer,\r\n#   -Clipped ReLU instead of PReLU (although PReLU may perform better)\r\n#   -the loss would be NLL of Connectionist Temporal Classification (CTC) cost\r\nmodel = Sequential()\r\nmodel.add(Dense(1520,2048))\r\nmodel.add(PReLU(2048))\r\nmodel.add(Dense(2048,2048))\r\nmodel.add(PReLU(2048))\r\nmodel.add(Dense(2048,2048))\r\nmodel.add(PReLU(2048))\r\nmodel.add(AdvancedReshape(new_shape_fn=lambda current_batch_size, current_shape: (current_batch_size, current_shape[0]/current_batch_size, current_shape[1])))\r\nmodel.add(SimpleRNN(2048, 2048))\r\nmodel.add(PReLU(2048))\r\nmodel.add(AdvancedReshape(new_shape_fn=lambda current_batch_size, current_shape: (current_batch_size*current_shape[1], current_shape[2])))\r\nmodel.add(Dense(2048,30, activation='softmax')\r\n\r\nsgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)\r\nmodel.compile(loss='categorical_crossentropy', optimizer='sgd')\r\n```"}}