{"action":"opened","issue":{"url":"https://api.github.com/repos/karpathy/char-rnn/issues/70","labels_url":"https://api.github.com/repos/karpathy/char-rnn/issues/70/labels{/name}","comments_url":"https://api.github.com/repos/karpathy/char-rnn/issues/70/comments","events_url":"https://api.github.com/repos/karpathy/char-rnn/issues/70/events","html_url":"https://github.com/karpathy/char-rnn/issues/70","id":97313211,"number":70,"title":"Gradient magnitute depends on seq_length","user":{"login":"apaszke","id":4583066,"avatar_url":"https://avatars.githubusercontent.com/u/4583066?v=3","gravatar_id":"","url":"https://api.github.com/users/apaszke","html_url":"https://github.com/apaszke","followers_url":"https://api.github.com/users/apaszke/followers","following_url":"https://api.github.com/users/apaszke/following{/other_user}","gists_url":"https://api.github.com/users/apaszke/gists{/gist_id}","starred_url":"https://api.github.com/users/apaszke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apaszke/subscriptions","organizations_url":"https://api.github.com/users/apaszke/orgs","repos_url":"https://api.github.com/users/apaszke/repos","events_url":"https://api.github.com/users/apaszke/events{/privacy}","received_events_url":"https://api.github.com/users/apaszke/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-07-26T12:41:07Z","updated_at":"2015-07-26T12:41:07Z","closed_at":null,"body":"Hi!\r\nI've used char-rnn as a base for a different learning task, which requires me to train the networks on much longer sequences. However I've quickly realized that the training is very unstable and requires a ridiculously small learning rate.\r\nAfter some fiddling I tried different sequence lengths and got the following results:\r\n\r\n| Sequence length | Gradient norm |\r\n| ----------------- | --------------- |\r\n| 10              | 3.00e+00      |\r\n| 250             | 6.11e+01      |\r\n| 500             | 1.05e+02      |\r\n\r\nApparently the gradient is growing approximately linearly with the sequence length.\r\nFortunately the fix is quite simple as it's enough to add: ```grad_params:div(opt.seq_length)``` at train.lua:267\r\nI'm not sure if it's also a problem for char-rnn's use case, but it seems logical, that the gradient should be of similar magnitude regardless of the ```seq_length```.\r\n\r\nAfter adding the line I'm getting the following results:\r\n\r\n| Sequence length | Gradient norm |\r\n|-----------------|---------------|\r\n| 10              | 2.93e-01      |\r\n| 250             | 2.85e-01      |\r\n| 500             | 3.00e-01      |"}}