{"action":"created","issue":{"url":"https://api.github.com/repos/airoldilab/sgd/issues/15","labels_url":"https://api.github.com/repos/airoldilab/sgd/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/airoldilab/sgd/issues/15/comments","events_url":"https://api.github.com/repos/airoldilab/sgd/issues/15/events","html_url":"https://github.com/airoldilab/sgd/issues/15","id":53981400,"number":15,"title":"P-dim Learning Rate Decreasing Too Fast","user":{"login":"k-ye","id":7481356,"avatar_url":"https://avatars.githubusercontent.com/u/7481356?v=3","gravatar_id":"","url":"https://api.github.com/users/k-ye","html_url":"https://github.com/k-ye","followers_url":"https://api.github.com/users/k-ye/followers","following_url":"https://api.github.com/users/k-ye/following{/other_user}","gists_url":"https://api.github.com/users/k-ye/gists{/gist_id}","starred_url":"https://api.github.com/users/k-ye/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/k-ye/subscriptions","organizations_url":"https://api.github.com/users/k-ye/orgs","repos_url":"https://api.github.com/users/k-ye/repos","events_url":"https://api.github.com/users/k-ye/events{/privacy}","received_events_url":"https://api.github.com/users/k-ye/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/airoldilab/sgd/labels/learning%20rate","name":"learning rate","color":"c7def8"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-01-11T04:27:53Z","updated_at":"2015-05-11T00:36:51Z","closed_at":null,"body":"Hi,\r\n\r\n@lantian2012 and I was doing some experiments on Poisson model dataset. We found out that if using `p-dim` learning rate, the learning rate will drop too fast. This is mainly because `Idiag` matrix is the summation of the square of `score_function` in each iteration. And this value will soon goes up to around 1e+18 (when X are around 10, theta.true is around [0.5, 1], Y ~ Pois(exp(theta' X))), which means its inversion being as tiny as 1e-18. At this point, the learning rate is almost 0, which means in each update, theta will only change for really a small value.\r\n\r\n@ptoulis Yesterday you said that we are calculating the square of the score function Gi^2 because this will approximate to the real Fisher Information Matrix. However, in our pset 3, it is listed that:\r\n` I(\\theta) = E(h'(x_n'theta) * x_n x_n') / phi`\r\n\r\nIt was using the expectation of the first derivative of transfer function (instead of the square of the score function). \r\n\r\nI've been trying to use something like `h'(x_n'theta)*(x_n x_n')` as learning rate. The result, though, was still disappointing."},"comment":{"url":"https://api.github.com/repos/airoldilab/sgd/issues/comments/100723818","html_url":"https://github.com/airoldilab/sgd/issues/15#issuecomment-100723818","issue_url":"https://api.github.com/repos/airoldilab/sgd/issues/15","id":100723818,"user":{"login":"dustinvtran","id":2569867,"avatar_url":"https://avatars.githubusercontent.com/u/2569867?v=3","gravatar_id":"","url":"https://api.github.com/users/dustinvtran","html_url":"https://github.com/dustinvtran","followers_url":"https://api.github.com/users/dustinvtran/followers","following_url":"https://api.github.com/users/dustinvtran/following{/other_user}","gists_url":"https://api.github.com/users/dustinvtran/gists{/gist_id}","starred_url":"https://api.github.com/users/dustinvtran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dustinvtran/subscriptions","organizations_url":"https://api.github.com/users/dustinvtran/orgs","repos_url":"https://api.github.com/users/dustinvtran/repos","events_url":"https://api.github.com/users/dustinvtran/events{/privacy}","received_events_url":"https://api.github.com/users/dustinvtran/received_events","type":"User","site_admin":false},"created_at":"2015-05-11T00:36:51Z","updated_at":"2015-05-11T00:36:51Z","body":"I spotted the error. The learning rate should not take into account any approximations to the Fisher information based on previous iterations. Each diagonal of the observed Fisher information is an approximation to `I(theta_n)` for different `n`. Hence it does not make sense to sum them up.\r\nTherefore\r\n```\r\nIdiag = Idiag + diag(Gi * Gi.transpose())\r\n```\r\nshould simply be \r\n```\r\nIdiag = diag(Gi * Gi.transpose())\r\n```"}}