{"action":"created","issue":{"url":"https://api.github.com/repos/benanne/Lasagne/issues/136","labels_url":"https://api.github.com/repos/benanne/Lasagne/issues/136/labels{/name}","comments_url":"https://api.github.com/repos/benanne/Lasagne/issues/136/comments","events_url":"https://api.github.com/repos/benanne/Lasagne/issues/136/events","html_url":"https://github.com/benanne/Lasagne/issues/136","id":57570579,"number":136,"title":"Additional updates for layers","user":{"login":"diogo149","id":1668093,"avatar_url":"https://avatars.githubusercontent.com/u/1668093?v=3","gravatar_id":"","url":"https://api.github.com/users/diogo149","html_url":"https://github.com/diogo149","followers_url":"https://api.github.com/users/diogo149/followers","following_url":"https://api.github.com/users/diogo149/following{/other_user}","gists_url":"https://api.github.com/users/diogo149/gists{/gist_id}","starred_url":"https://api.github.com/users/diogo149/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/diogo149/subscriptions","organizations_url":"https://api.github.com/users/diogo149/orgs","repos_url":"https://api.github.com/users/diogo149/repos","events_url":"https://api.github.com/users/diogo149/events{/privacy}","received_events_url":"https://api.github.com/users/diogo149/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-02-13T08:07:35Z","updated_at":"2015-02-13T21:40:27Z","closed_at":null,"body":"While working on https://github.com/benanne/Lasagne/issues/133, I realized that there isn't a way for a layer to update it's state outside of the normal `updates.py` updates. Specifically, weight decay for learnable activation units and rolling averages for batch normalization.\r\n\r\nOne proposed solution would be to have some additional `get_additional_updates` method, like `get_params`, that gets merged into the updates. Implementation should be pretty clean/straightforward if one is careful to merge the updates together (since each var can only be updated once, and you don't want to double add the var)."},"comment":{"url":"https://api.github.com/repos/benanne/Lasagne/issues/comments/74331698","html_url":"https://github.com/benanne/Lasagne/issues/136#issuecomment-74331698","issue_url":"https://api.github.com/repos/benanne/Lasagne/issues/136","id":74331698,"user":{"login":"benanne","id":43829,"avatar_url":"https://avatars.githubusercontent.com/u/43829?v=3","gravatar_id":"","url":"https://api.github.com/users/benanne","html_url":"https://github.com/benanne","followers_url":"https://api.github.com/users/benanne/followers","following_url":"https://api.github.com/users/benanne/following{/other_user}","gists_url":"https://api.github.com/users/benanne/gists{/gist_id}","starred_url":"https://api.github.com/users/benanne/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benanne/subscriptions","organizations_url":"https://api.github.com/users/benanne/orgs","repos_url":"https://api.github.com/users/benanne/repos","events_url":"https://api.github.com/users/benanne/events{/privacy}","received_events_url":"https://api.github.com/users/benanne/received_events","type":"User","site_admin":false},"created_at":"2015-02-13T21:40:27Z","updated_at":"2015-02-13T21:40:27Z","body":"> I also was thinking of having layers be responsible for updating themselves. An initial thought would be a `get_all_updates`, similar to `get_all_params`, which recursively gets the updates, and layers could be passed in an update function to use (eg. `updates.momentum`) either explicitly or recursively from a different layer.\r\n\r\nWhat would be the benefit of that? To me it sounds like it would be a lot more complex and somewhat counterintuitive (networks don't learn by themselves, an optimization algorithm 'teaches' them and this is external to the network). If we decide to have layers generate updates themselves, I think this should be restricted to updates that are not part of the optimization algorithm (like the rolling averages in batch normalization for example)."}}