{"action":"created","issue":{"url":"https://api.github.com/repos/mila-udem/blocks/issues/851","labels_url":"https://api.github.com/repos/mila-udem/blocks/issues/851/labels{/name}","comments_url":"https://api.github.com/repos/mila-udem/blocks/issues/851/comments","events_url":"https://api.github.com/repos/mila-udem/blocks/issues/851/events","html_url":"https://github.com/mila-udem/blocks/pull/851","id":109382527,"number":851,"title":"WIP: Batch normalization","user":{"login":"cooijmanstim","id":206920,"avatar_url":"https://avatars.githubusercontent.com/u/206920?v=3","gravatar_id":"","url":"https://api.github.com/users/cooijmanstim","html_url":"https://github.com/cooijmanstim","followers_url":"https://api.github.com/users/cooijmanstim/followers","following_url":"https://api.github.com/users/cooijmanstim/following{/other_user}","gists_url":"https://api.github.com/users/cooijmanstim/gists{/gist_id}","starred_url":"https://api.github.com/users/cooijmanstim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cooijmanstim/subscriptions","organizations_url":"https://api.github.com/users/cooijmanstim/orgs","repos_url":"https://api.github.com/users/cooijmanstim/repos","events_url":"https://api.github.com/users/cooijmanstim/events{/privacy}","received_events_url":"https://api.github.com/users/cooijmanstim/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-10-01T20:57:58Z","updated_at":"2015-10-01T23:14:06Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/mila-udem/blocks/pulls/851","html_url":"https://github.com/mila-udem/blocks/pull/851","diff_url":"https://github.com/mila-udem/blocks/pull/851.diff","patch_url":"https://github.com/mila-udem/blocks/pull/851.patch"},"body":"This continues the work done on #513. Current status:\r\n\r\nGammas/betas no longer need to be provided by the user, but the default initialization can be overridden with the given_gammas/given_betas kwargs. The initial population statistics can be similarly specified. These keyword arguments should be dicts with variables for keys and initial values for values.\r\n\r\nThe previous version of apply_batch_normalization had a use_population keyword argument that allowed per-variable switching between batch/population statistics. If necessary, this can now be done by multiple calls to apply_batch_normalization.\r\n\r\nTo get from here to a fully functional BN implementation, we need to think about how to save/load the BN parameters. The gammas and betas are already recognized by blocks as parameters, and so the user could construct the given_gammas/given_betas argument from a saved model's parameter dictionary (maybe a helper function for this would be convenient). The population statistics however aren't saved/loaded yet when they definitely should be as they are part of the knowledge the model has acquired. I am open to suggestions on how to do this.\r\n\r\nThe gist at https://gist.github.com/koningrobot/8b570f38cbb174ab0438 shows an example of the new usage.\r\n"},"comment":{"url":"https://api.github.com/repos/mila-udem/blocks/issues/comments/144874038","html_url":"https://github.com/mila-udem/blocks/pull/851#issuecomment-144874038","issue_url":"https://api.github.com/repos/mila-udem/blocks/issues/851","id":144874038,"user":{"login":"dwf","id":60206,"avatar_url":"https://avatars.githubusercontent.com/u/60206?v=3","gravatar_id":"","url":"https://api.github.com/users/dwf","html_url":"https://github.com/dwf","followers_url":"https://api.github.com/users/dwf/followers","following_url":"https://api.github.com/users/dwf/following{/other_user}","gists_url":"https://api.github.com/users/dwf/gists{/gist_id}","starred_url":"https://api.github.com/users/dwf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dwf/subscriptions","organizations_url":"https://api.github.com/users/dwf/orgs","repos_url":"https://api.github.com/users/dwf/repos","events_url":"https://api.github.com/users/dwf/events{/privacy}","received_events_url":"https://api.github.com/users/dwf/received_events","type":"User","site_admin":false},"created_at":"2015-10-01T23:14:06Z","updated_at":"2015-10-01T23:14:06Z","body":"@cooijmanstim Thanks for picking this up.\r\n\r\n> we need to think about how to save/load the BN parameters. \r\n\r\nBy \"saved\" you mean serialized with a checkpoint? It seems like where to save that entire test-time graph is more the issue, isn't it? One would like to be able to get access to and evaluate it.\r\n\r\nOne thing I think we should leave up to the user is whether to adapt these via some additional squared error objective function passed as a term of the cost function to GradientDescent (which seems like a straightfoward and easy way to do it) or just via some simple moving average updates added with `add_updates`. In either case I don't think it's a problem to consider these quantities PARAMETERs as they don't appear in the train-time graph and therefore won't be adapted if you do gradient descent on the train-time graph's (single) output.\r\n\r\nThere's a larger arc to this story which is whether the graph substitution framework is really appropriate for this situation. Other frameworks are tackling this by adding a \"Brick\" equivalent into the original model object hierarchy, which makes it a little easier to figure out what's going on. In that kind of a setup, from a  Blocks perspective, we'd essentially have two apply methods, one for train time and one for test time. In that situation, however, we'd need some way of conveniently excluding the learned means and variances from gradient descent on the real objective function, which probably means a new parent role for PARAMETER, like ADAPTABLE or something."}}