{"action":"created","issue":{"url":"https://api.github.com/repos/nengo/nengo/issues/643","labels_url":"https://api.github.com/repos/nengo/nengo/issues/643/labels{/name}","comments_url":"https://api.github.com/repos/nengo/nengo/issues/643/comments","events_url":"https://api.github.com/repos/nengo/nengo/issues/643/events","html_url":"https://github.com/nengo/nengo/issues/643","id":56583396,"number":643,"title":"Should we normalize learning rate by # neurons?","user":{"login":"arvoelke","id":5420057,"avatar_url":"https://avatars.githubusercontent.com/u/5420057?v=3","gravatar_id":"","url":"https://api.github.com/users/arvoelke","html_url":"https://github.com/arvoelke","followers_url":"https://api.github.com/users/arvoelke/followers","following_url":"https://api.github.com/users/arvoelke/following{/other_user}","gists_url":"https://api.github.com/users/arvoelke/gists{/gist_id}","starred_url":"https://api.github.com/users/arvoelke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arvoelke/subscriptions","organizations_url":"https://api.github.com/users/arvoelke/orgs","repos_url":"https://api.github.com/users/arvoelke/repos","events_url":"https://api.github.com/users/arvoelke/events{/privacy}","received_events_url":"https://api.github.com/users/arvoelke/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/nengo/nengo/labels/discussion","name":"discussion","color":"207de5"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":15,"created_at":"2015-02-04T20:33:59Z","updated_at":"2015-02-05T03:22:02Z","closed_at":null,"body":"Amir noticed with the two-link arm that if you crank up the number of neurons, unintuitively the performance degrades. \r\n\r\nWhat is going on is that the magnitude of each decoder gets smaller the more neurons you have, but the learning deltas remain the same. Applying the relatively larger delta makes the learning oscillate. Instead, I think the delta should be scaled by the average size of each decoder, which Eric and I believe is just inversely proportional to the number of neurons.\r\n\r\nSo I propose we scale the learning rate by dividing by `n_neurons`. This will also make the number \"friendlier\", since it is currently on the order of `1e-6` for 500 neurons, and so the change would make it `5e-4`."},"comment":{"url":"https://api.github.com/repos/nengo/nengo/issues/comments/72987883","html_url":"https://github.com/nengo/nengo/issues/643#issuecomment-72987883","issue_url":"https://api.github.com/repos/nengo/nengo/issues/643","id":72987883,"user":{"login":"studywolf","id":2473097,"avatar_url":"https://avatars.githubusercontent.com/u/2473097?v=3","gravatar_id":"","url":"https://api.github.com/users/studywolf","html_url":"https://github.com/studywolf","followers_url":"https://api.github.com/users/studywolf/followers","following_url":"https://api.github.com/users/studywolf/following{/other_user}","gists_url":"https://api.github.com/users/studywolf/gists{/gist_id}","starred_url":"https://api.github.com/users/studywolf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/studywolf/subscriptions","organizations_url":"https://api.github.com/users/studywolf/orgs","repos_url":"https://api.github.com/users/studywolf/repos","events_url":"https://api.github.com/users/studywolf/events{/privacy}","received_events_url":"https://api.github.com/users/studywolf/received_events","type":"User","site_admin":false},"created_at":"2015-02-05T03:22:01Z","updated_at":"2015-02-05T03:22:01Z","body":"I feel pretty strongly against any hidden scaling of the learning rate, how would this be done? Maybe a parameter you could set to true would be more appropriate for this than automatically doing it? otherwise it might be trying to do too much for the user automatically...and yeah different learning rules like terry pointed out! it gets complicated!"}}