{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/838","labels_url":"https://api.github.com/repos/fchollet/keras/issues/838/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/838/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/838/events","html_url":"https://github.com/fchollet/keras/issues/838","id":111568746,"number":838,"title":"Discussing New APIs","user":{"login":"dbonadiman","id":1776151,"avatar_url":"https://avatars.githubusercontent.com/u/1776151?v=3","gravatar_id":"","url":"https://api.github.com/users/dbonadiman","html_url":"https://github.com/dbonadiman","followers_url":"https://api.github.com/users/dbonadiman/followers","following_url":"https://api.github.com/users/dbonadiman/following{/other_user}","gists_url":"https://api.github.com/users/dbonadiman/gists{/gist_id}","starred_url":"https://api.github.com/users/dbonadiman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dbonadiman/subscriptions","organizations_url":"https://api.github.com/users/dbonadiman/orgs","repos_url":"https://api.github.com/users/dbonadiman/repos","events_url":"https://api.github.com/users/dbonadiman/events{/privacy}","received_events_url":"https://api.github.com/users/dbonadiman/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-10-15T08:32:31Z","updated_at":"2015-12-30T13:46:21Z","closed_at":"2015-11-18T14:32:42Z","body":"Hello everyone,\r\nI open this post to present some new APIs and types of model i have in mind, before start working on them in order to understand: are they feasible? are they needed? or simply understand if it is the best way to do this stuff.\r\n\r\n### Nesting Models\r\nEdit: Just realised that under certain circumstance that is actually possible using: [Containers](https://github.com/fchollet/keras/blob/master/keras/layers/containers.py)\r\n\r\nThe first idea is about nesting models. Models and layer have in common the fact that both of them takes an array or a tensor in input and produces another tensor as output so it should be feasible to do something like:\r\n\r\n```python\r\nseq1 = Sequential()\r\nseq1.add(Dense(10))\r\nseq1.add(Activation('relu'))\r\nseq1.add(Dense(10))\r\nseq1.add(Activation('relu'))\r\n\r\nseq2 = Sequential()\r\nseq2.add(Dense(40))\r\nseq2.add(Activation('relu'))\r\nseq2.add(Dense(10))\r\nseq2.add(Activation('relu'))\r\n\r\nmodel = Graph()\r\nmodel.add_input(name='input1', input_shape=(10,))\r\nmodel.add_input(name='input2', input_shape=(40,))\r\nmodel.add_node(seq1, name='sequential1', input='input1')\r\nmodel.add_node(seq2, name='sequential2', input='input2')\r\nmodel.add_node(Dense(50), name='out', inputs=['sequential1','sequential2'])\r\nmodel.add_output(name = 'output', input ='out')\r\n```\r\n\r\nAt the current stage it is possible to implement the same model completely in the Graph model so this is an API change that only changes how the model presents itself visually. I presented this changes because it allows another possibility that i will highlight in point 2.\r\n\r\n### TimeDistributedModels\r\nThe only way to deal with variable size sequences in Keras at the moment is by using recurrent neural networks or pad the input sequences until (that sometimes is quite problematic). But in practices it is theoretically possible to deal with variables size inputs in a simpler way that sometimes is effective at least as recurrent networks, the sliding window model. \r\nHere is a simple example, take a sentence in the form:\r\n*Keras is a powerful framework*\r\nAnd the task is to assign a tag for each word, for example a model that spot the word Keras (A really dumb task to be honest).\r\nso the sentence is converted as array of indexes:\r\n```\r\n[1, 2, 3, 4, 5]\r\n```\r\nNow we can feed it one word at the time to our model and predict the output as binary.\r\nso we **need** to split it in vector of dimension 1 and feed it to the network one word at the time\r\n```python\r\nmodel = Sequential()\r\nmodel.add(Embedding(10, 50) input_lenght=1)\r\nmodel.add(...)\r\n...\r\nmodel.add(...)\r\nmodel.add(Activation('sigmoid'))\r\n```\r\nnow assume that to spot Keras we need at least one word on the left and one word on the right of the target word. So we pad the sequence.\r\n```\r\n[0, 1, 2, 3, 4, 5, 0]\r\n```\r\nand convert it to a to dimensional input\r\n\r\n```\r\n[[0, 1, 2], [1, 2, 3], ..., [4, 5, 0]]\r\n```\r\nagain each of this vector needs to be fed one by one to the network.\r\nNow assume that we have this new kind of model lets call it TimeDistributedModel and that the API modification in point 1 is implemented.\r\nWe can simply:\r\n```python\r\ntdmodel = TimeDistributed(time_axis=1)\r\ntdmodel.add(model)\r\n```\r\nThat will simply apply the model defined above over a time dimension and produces at each step a sequence.\r\nIn the case the point 1 is not implemented it is possible to add the layer directly to the model. \r\n\r\n```python\r\nmodel = TimeDistributed(time_axis=1)\r\nmodel.add(Embedding(10, 50) input_lenght=1)\r\nmodel.add(...)\r\n...\r\nmodel.add(...)\r\nmodel.add(Activation('sigmoid'))\r\n```\r\n\r\n### Unleash the power of Convolutions\r\nThe sliding window model it is not the only kind of model that is used to deal with the variable size of sentence but even convolutional ones. We can implement that is by implementing the k-MaxPooling layer that differs from MaxPooling1D for the fact that all the output it is not subsampled by a factor but the k-Max values for each filter are taken, by doing that the output of a convolution-maxpooling (of any size) is (k, nb_filters) and do not depends on the length of the input sequence.\r\n\r\n### What these modification allows me to do in practice?\r\nBelow i will post some papers that would be easily implemented given the discussed modification.\r\n[NLP (almost) from scratch](http://arxiv.org/abs/1103.0398)\r\n[Learning Character-level Representations for Part-of-Speech Tagging](http://jmlr.org/proceedings/papers/v32/santos14.pdf)\r\n[A Convolutional Neural Network for Modelling Sentences](http://nal.co/papers/Kalchbrenner_DCNN_ACL14)\r\n\r\nWhat do you think? Is it feasible to some extend?\r\n"},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/168001224","html_url":"https://github.com/fchollet/keras/issues/838#issuecomment-168001224","issue_url":"https://api.github.com/repos/fchollet/keras/issues/838","id":168001224,"user":{"login":"man2code","id":10802530,"avatar_url":"https://avatars.githubusercontent.com/u/10802530?v=3","gravatar_id":"","url":"https://api.github.com/users/man2code","html_url":"https://github.com/man2code","followers_url":"https://api.github.com/users/man2code/followers","following_url":"https://api.github.com/users/man2code/following{/other_user}","gists_url":"https://api.github.com/users/man2code/gists{/gist_id}","starred_url":"https://api.github.com/users/man2code/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/man2code/subscriptions","organizations_url":"https://api.github.com/users/man2code/orgs","repos_url":"https://api.github.com/users/man2code/repos","events_url":"https://api.github.com/users/man2code/events{/privacy}","received_events_url":"https://api.github.com/users/man2code/received_events","type":"User","site_admin":false},"created_at":"2015-12-30T13:46:21Z","updated_at":"2015-12-30T13:46:21Z","body":"I believe a first step towards implementing TimeDistributedModels would be not to fix the input dimensions in the layers' initialization/building, as the hidden layer on theano does (http://deeplearning.net/tutorial/mlp.html). I'm not sure though the implications of doing that to the whole package."}}