{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/224","labels_url":"https://api.github.com/repos/fchollet/keras/issues/224/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/224/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/224/events","html_url":"https://github.com/fchollet/keras/issues/224","id":88034243,"number":224,"title":"Split (or multiple objective functions)","user":{"login":"jgbos","id":2148522,"avatar_url":"https://avatars.githubusercontent.com/u/2148522?v=3","gravatar_id":"","url":"https://api.github.com/users/jgbos","html_url":"https://github.com/jgbos","followers_url":"https://api.github.com/users/jgbos/followers","following_url":"https://api.github.com/users/jgbos/following{/other_user}","gists_url":"https://api.github.com/users/jgbos/gists{/gist_id}","starred_url":"https://api.github.com/users/jgbos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jgbos/subscriptions","organizations_url":"https://api.github.com/users/jgbos/orgs","repos_url":"https://api.github.com/users/jgbos/repos","events_url":"https://api.github.com/users/jgbos/events{/privacy}","received_events_url":"https://api.github.com/users/jgbos/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-06-13T17:03:25Z","updated_at":"2015-06-21T08:55:12Z","closed_at":null,"body":"Apologies if this has already been asked, couldn't find anything though.  But do you have the capability to utilize multiple objective functions, say a \"Split\" similar to \"Merge\"?  The goal would be to implement a model like the one given here:\r\n\r\nhttps://github.com/rbgirshick/fast-rcnn\r\n\r\nThat is, have a classifier and a regression objective function that both provide gradient information to backprop.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/113878828","html_url":"https://github.com/fchollet/keras/issues/224#issuecomment-113878828","issue_url":"https://api.github.com/repos/fchollet/keras/issues/224","id":113878828,"user":{"login":"pdermyer","id":8903716,"avatar_url":"https://avatars.githubusercontent.com/u/8903716?v=3","gravatar_id":"","url":"https://api.github.com/users/pdermyer","html_url":"https://github.com/pdermyer","followers_url":"https://api.github.com/users/pdermyer/followers","following_url":"https://api.github.com/users/pdermyer/following{/other_user}","gists_url":"https://api.github.com/users/pdermyer/gists{/gist_id}","starred_url":"https://api.github.com/users/pdermyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pdermyer/subscriptions","organizations_url":"https://api.github.com/users/pdermyer/orgs","repos_url":"https://api.github.com/users/pdermyer/repos","events_url":"https://api.github.com/users/pdermyer/events{/privacy}","received_events_url":"https://api.github.com/users/pdermyer/received_events","type":"User","site_admin":false},"created_at":"2015-06-21T08:55:12Z","updated_at":"2015-06-21T08:55:12Z","body":"I think the crux is that whatever the syntax, we need to end up with a single object providing the model interface for training and prediction.\r\n\r\nThe Fork Layer:\r\nThe fork layer above is awkward. It relies on the add() function of the underlying model to specify the position, which should be a concern of the fork object. It falls apart when we consider the case with more than one fork in a model. In which position for which fork are we specifying when we call add() on the model?\r\n\r\nThe Fork Model:\r\nThe fork model more naturally accommodates multiple forking. But when you look at complex models, you see a mess of forking and multiple auxiliary classifiers (see GoogLeNet http://arxiv.org/pdf/1409.4842.pdf). In such a case, clearly there isn't going to be just one fork model that we can interact with for training, so we wont end up with a final manageable, trainable model. In the simple case, however, the fork model in #104 seems the best option so far.\r\n\r\nTo distill the requirements as I understand them:  We would like to take n input layers (multi-modal) to m output layers with at most one (weighted) objective function applied to each of these m output layers. We should be able to compile and fit the model in the same way as the sequential model. \r\n\r\nTo that end, I wonder if the most sane route might be adding a graph model like what I sketch below:\r\n\r\n```python\r\n#sub models like text_model become immutable after fork or merge to prevent surprises.\r\nmodel_graph = Graph()\r\n\r\ntext_model = model_graph.create_input()\r\ntext_model.add(…)\r\ntext_model.add(…)\r\n…\r\n\r\nimage_model = model_graph.create_input()\r\nimage_model.add(…)\r\nimage_model.add(…)\r\n…\r\n\r\n#adding auxiliary classification\r\nimage_model, image_intermediate = model_graph.fork(image_model, n=2)\r\nimage_intermediate.add(Dense(4096,1000))\r\nimage_intermediate.add(Activation(‘softmax’))\r\nmodel_graph.create_output(image_intermediate, objective1)\r\n\r\n#merging models\r\ntext_and_image_model = model_graph.merge( (text_model, image_model), mode=‘concat’)\r\ntext_and_image_model.add(…)\r\ntext_and_image_model.add(…)\r\n\r\nmodel_graph.create_output(text_and_image_model, objective2)\r\n\r\nmodel_graph.compile(optimizer)\r\nmodel_graph.fit([X_text, X_image], [y_image, y_final])\r\n```"}}