{"action":"opened","issue":{"url":"https://api.github.com/repos/nicholas-leonard/rnn/issues/3","labels_url":"https://api.github.com/repos/nicholas-leonard/rnn/issues/3/labels{/name}","comments_url":"https://api.github.com/repos/nicholas-leonard/rnn/issues/3/comments","events_url":"https://api.github.com/repos/nicholas-leonard/rnn/issues/3/events","html_url":"https://github.com/nicholas-leonard/rnn/issues/3","id":66987888,"number":3,"title":"Growing weights","user":{"login":"pefi9","id":11693349,"avatar_url":"https://avatars.githubusercontent.com/u/11693349?v=3","gravatar_id":"","url":"https://api.github.com/users/pefi9","html_url":"https://github.com/pefi9","followers_url":"https://api.github.com/users/pefi9/followers","following_url":"https://api.github.com/users/pefi9/following{/other_user}","gists_url":"https://api.github.com/users/pefi9/gists{/gist_id}","starred_url":"https://api.github.com/users/pefi9/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pefi9/subscriptions","organizations_url":"https://api.github.com/users/pefi9/orgs","repos_url":"https://api.github.com/users/pefi9/repos","events_url":"https://api.github.com/users/pefi9/events{/privacy}","received_events_url":"https://api.github.com/users/pefi9/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-04-07T20:52:18Z","updated_at":"2015-04-07T20:52:18Z","closed_at":null,"body":"Hi,\r\n\r\nthanks for great recurrent package. I`m still new to Torch, so the problem is probably in my usage. However, when I try to stack LSTM into multilayer network by using Sequencer, the weights are constantly growing. Without sequencer with only one recurrent layer it works fine.\r\n\r\nnetwork:\r\n-- LSTM\r\n    recurrent = nn.LSTM(inSize, hiddenSize1, rho)\r\n    recurrent2 = nn.LSTM(hiddenSize1, hiddenSize2, rho)\r\n    recurrent.scales = torch.Tensor(rho):fill(1)\r\n    recurrent2.scales = torch.Tensor(rho):fill(1)\r\n    linear = nn.Linear(hiddenSize2, outSize)\r\n\r\n    model = nn.Sequential()\r\n    sequencer = nn.Sequencer(recurrent)\r\n    sequencer2 = nn.Sequencer(recurrent2)\r\n    sequencer3 = nn.Sequencer(linear)\r\n    model:add(sequencer)\r\n    model:add(sequencer2)\r\n    model:add(sequencer3)\r\n\r\n-- TRAINING\r\n\r\nfor t = 1, (trainSize - batchSize) do\r\n\r\n        -- load new sample\r\n        local inputs, targets, gradOutputs = {}, {}, {}\r\n        for step = 1, rho do\r\n            local index = t + step\r\n            inputs[step] = inputData:sub(1, inputData:size(1), index, index + batchSize - 1):transpose(1,2)\r\n            targets[step] = outputData:sub(1, outputData:size(1), index, index + batchSize - 1):transpose(1, 2)\r\n\r\n        end\r\n\r\n        local outputs = model:forward(inputs)\r\n\r\n        for step = 1, rho do\r\n            local err = criterion:forward(outputs[step], targets[step])\r\n            trainError = trainError + err\r\n            gradOutputs[step] = criterion:backward(outputs[step], targets[step])\r\n        end\r\n\r\n        model:backward(inputs, gradOutputs)\r\n        model:updateParameters(lrRNN)\r\n        linear:zeroGradParameters()\r\n    end\r\n\r\nI have tried lowering the learning rate (1e-2, 1e-3, 1e-4), different size of scales (1, 1/rho), batch size (1,10) or rho (1,5,10,100). But none of these modification seemed to work. Any ideas?"}}