{"action":"created","issue":{"url":"https://api.github.com/repos/casperkaae/parmesan/issues/15","labels_url":"https://api.github.com/repos/casperkaae/parmesan/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/casperkaae/parmesan/issues/15/comments","events_url":"https://api.github.com/repos/casperkaae/parmesan/issues/15/events","html_url":"https://github.com/casperkaae/parmesan/issues/15","id":111805478,"number":15,"title":"BatchNormalization","user":{"login":"Botev","id":1889878,"avatar_url":"https://avatars.githubusercontent.com/u/1889878?v=3","gravatar_id":"","url":"https://api.github.com/users/Botev","html_url":"https://github.com/Botev","followers_url":"https://api.github.com/users/Botev/followers","following_url":"https://api.github.com/users/Botev/following{/other_user}","gists_url":"https://api.github.com/users/Botev/gists{/gist_id}","starred_url":"https://api.github.com/users/Botev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Botev/subscriptions","organizations_url":"https://api.github.com/users/Botev/orgs","repos_url":"https://api.github.com/users/Botev/repos","events_url":"https://api.github.com/users/Botev/events{/privacy}","received_events_url":"https://api.github.com/users/Botev/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-10-16T10:37:18Z","updated_at":"2015-10-16T13:53:08Z","closed_at":null,"body":"It would be nice if the BatchNormalizationLayer, rather than supporting a \"single_pass\" would support \"collect\" where it will collect the variables in minibatches. If the dataset is big enough the \"single_pass\" could fail. Alternative would be something like this:\r\n```\r\nif collect:\r\n                # This will collect the dataset statistics on minibatches\r\n                # However to this accurately we will need a new extra variable\r\n                # for E[x^2] and use its average instead of the std\r\n                running_ex2 = theano.shared(np.zeros_like(self.std.get_value()))\r\n                t = theano.shared(1)\r\n                ex2 = input.sqr().mean(self.axes, keepdims=True)\r\n                mean_update = ((t-1) / t) * running_mean + mean / t\r\n                ex2_update = ((t-1) / t) * running_ex2 + ex2 / t\r\n                std_update = (ex2_update - mean_update.sqr() + self.epsilon).sqrt()\r\n                # Set the default updates\r\n                running_mean.default_update = mean_update\r\n                running_ex2.default_update = ex2_update\r\n                running_std.default_update = std_update\r\n                t.default_update = t + 1\r\n                # and include them in the graph so their default updates will be\r\n                # applied (although the expressions will be optimized away later)\r\n                mean += 0 * running_mean + 0 * t\r\n                std += 0 * running_std + 0 * running_ex2\r\n            else:\r\n                # During training instead we use a geometric moving average\r\n                running_mean.default_update = ((1 - self.alpha) * running_mean +\r\n                                               self.alpha * mean)\r\n                running_std.default_update = ((1 - self.alpha) * running_std +\r\n                                              self.alpha * std)\r\n                # and include them in the graph so their default updates will be\r\n                # applied (although the expressions will be optimized away later)\r\n                mean += 0 * running_mean\r\n                std += 0 * running_std\r\n```"},"comment":{"url":"https://api.github.com/repos/casperkaae/parmesan/issues/comments/148722541","html_url":"https://github.com/casperkaae/parmesan/issues/15#issuecomment-148722541","issue_url":"https://api.github.com/repos/casperkaae/parmesan/issues/15","id":148722541,"user":{"login":"skaae","id":2623134,"avatar_url":"https://avatars.githubusercontent.com/u/2623134?v=3","gravatar_id":"","url":"https://api.github.com/users/skaae","html_url":"https://github.com/skaae","followers_url":"https://api.github.com/users/skaae/followers","following_url":"https://api.github.com/users/skaae/following{/other_user}","gists_url":"https://api.github.com/users/skaae/gists{/gist_id}","starred_url":"https://api.github.com/users/skaae/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/skaae/subscriptions","organizations_url":"https://api.github.com/users/skaae/orgs","repos_url":"https://api.github.com/users/skaae/repos","events_url":"https://api.github.com/users/skaae/events{/privacy}","received_events_url":"https://api.github.com/users/skaae/received_events","type":"User","site_admin":false},"created_at":"2015-10-16T13:53:07Z","updated_at":"2015-10-16T13:53:07Z","body":"Single pass assumes that you pass the entire dataset through the network in a single batch. That way you collect the correct statistics for all batch normalization layers. \r\n\r\nIt's probably correct that `alpha!='single_pass'` uses a geometric average. Its copied from @f0k 's implementation. "}}