{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/211","labels_url":"https://api.github.com/repos/fchollet/keras/issues/211/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/211/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/211/events","html_url":"https://github.com/fchollet/keras/issues/211","id":86899702,"number":211,"title":"Feedback from callbacks","user":{"login":"tdhd","id":8267636,"avatar_url":"https://avatars.githubusercontent.com/u/8267636?v=3","gravatar_id":"","url":"https://api.github.com/users/tdhd","html_url":"https://github.com/tdhd","followers_url":"https://api.github.com/users/tdhd/followers","following_url":"https://api.github.com/users/tdhd/following{/other_user}","gists_url":"https://api.github.com/users/tdhd/gists{/gist_id}","starred_url":"https://api.github.com/users/tdhd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdhd/subscriptions","organizations_url":"https://api.github.com/users/tdhd/orgs","repos_url":"https://api.github.com/users/tdhd/repos","events_url":"https://api.github.com/users/tdhd/events{/privacy}","received_events_url":"https://api.github.com/users/tdhd/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-06-10T09:15:37Z","updated_at":"2015-06-12T22:56:36Z","closed_at":"2015-06-12T22:56:36Z","body":"Hey François,\r\n\r\nI think it would be useful to have the callbacks return an optional feedback to the `fit` method. This would be useful to let the callbacks control in certain aspects how `fit` behaves.\r\n\r\nFor example when thinking of an early stopping callback, it could tell `fit` to stop gracefully by exiting the loop over the epochs. Yesterday I was reading over http://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf, where they also let the time to train a model be a factor in their optimization. So being able to stop the training early would have a benefit for hyper parameter optimization.\r\n\r\nIf you look at my current implementation, it calls `sys.exit` in which case the overall optimization of hyper parameters would only resume when the training has been spawned in a different thread:\r\n\r\n```python\r\nclass EarlyStop(Callback):\r\n    def __init__(self, nb_epoch_lookback=5, verbose=0):\r\n        super(Callback, self).__init__()\r\n\r\n        self.nb_epoch_lookback = nb_epoch_lookback\r\n        self.verbose = verbose\r\n        self.best_val_loss_epoch = 0\r\n        self.best_val_loss = np.Inf\r\n\r\n    def on_epoch_end(self, epoch, logs={}):\r\n        '''currently, on_epoch_end receives epoch_logs from keras.models.Sequential.fit\r\n        which does only contain, if at all, the validation loss and validation accuracy'''\r\n        if self.params['do_validation']:\r\n            cur_val_loss = logs.get('val_loss')\r\n            if cur_val_loss < self.best_val_loss:\r\n                self.best_val_loss_epoch = epoch\r\n                self.best_val_loss = cur_val_loss\r\n            elif (epoch - self.best_val_loss_epoch) > self.nb_epoch_lookback:\r\n                if self.verbose > 0:\r\n                    print(\"EarlyStop: Did not observe an improvement over the last {0} epochs, stopping.\".format(self.nb_epoch_lookback))\r\n                import sys\r\n                sys.exit(0)\r\n        else:\r\n            import warnings\r\n            warnings.warn(\"Can run EarlyStop callback only with validation data, skipping\", RuntimeWarning)\r\n```\r\n\r\nUsing the feedback approach, the hyper parameter optimization could continue, regardless of whether running the training processes in different threads.\r\n\r\nConcretely the feedback could be implemented as an aggregate in `CallbackList` where all the logic of feedback aggregation is implemented in `CallbackList` and a single feedback is returned. Alternatively a list of feedbacks could be returned to `fit`.\r\n\r\nLet me know what you think about it."},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/111636397","html_url":"https://github.com/fchollet/keras/issues/211#issuecomment-111636397","issue_url":"https://api.github.com/repos/fchollet/keras/issues/211","id":111636397,"user":{"login":"tdhd","id":8267636,"avatar_url":"https://avatars.githubusercontent.com/u/8267636?v=3","gravatar_id":"","url":"https://api.github.com/users/tdhd","html_url":"https://github.com/tdhd","followers_url":"https://api.github.com/users/tdhd/followers","following_url":"https://api.github.com/users/tdhd/following{/other_user}","gists_url":"https://api.github.com/users/tdhd/gists{/gist_id}","starred_url":"https://api.github.com/users/tdhd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdhd/subscriptions","organizations_url":"https://api.github.com/users/tdhd/orgs","repos_url":"https://api.github.com/users/tdhd/repos","events_url":"https://api.github.com/users/tdhd/events{/privacy}","received_events_url":"https://api.github.com/users/tdhd/received_events","type":"User","site_admin":false},"created_at":"2015-06-12T22:56:35Z","updated_at":"2015-06-12T22:56:35Z","body":"Closing this, discussion in #219 "}}