{"action":"created","issue":{"url":"https://api.github.com/repos/karpathy/char-rnn/issues/38","labels_url":"https://api.github.com/repos/karpathy/char-rnn/issues/38/labels{/name}","comments_url":"https://api.github.com/repos/karpathy/char-rnn/issues/38/comments","events_url":"https://api.github.com/repos/karpathy/char-rnn/issues/38/events","html_url":"https://github.com/karpathy/char-rnn/issues/38","id":88237100,"number":38,"title":"train_loss and grad/param norm always are nan","user":{"login":"Taschenschieber","id":5377412,"avatar_url":"https://avatars.githubusercontent.com/u/5377412?v=3","gravatar_id":"","url":"https://api.github.com/users/Taschenschieber","html_url":"https://github.com/Taschenschieber","followers_url":"https://api.github.com/users/Taschenschieber/followers","following_url":"https://api.github.com/users/Taschenschieber/following{/other_user}","gists_url":"https://api.github.com/users/Taschenschieber/gists{/gist_id}","starred_url":"https://api.github.com/users/Taschenschieber/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Taschenschieber/subscriptions","organizations_url":"https://api.github.com/users/Taschenschieber/orgs","repos_url":"https://api.github.com/users/Taschenschieber/repos","events_url":"https://api.github.com/users/Taschenschieber/events{/privacy}","received_events_url":"https://api.github.com/users/Taschenschieber/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":66,"created_at":"2015-06-14T19:40:17Z","updated_at":"2015-07-09T08:50:35Z","closed_at":null,"body":"I'm running your current version on Antergos Linux x64. This happens when I try and start training on the (unmodified) tinyshakespeare model:\r\n\r\n```\r\n[stephan@antergos char-rnn]$ th train.lua\r\npackage cunn not found!\t\r\npackage cutorch not found!\t\r\nFalling back on CPU mode\t\r\nloading data files...\t\r\ncutting off end of data so that the batches/sequences divide evenly\t\r\nreshaping tensor...\t\r\ndata load done. Number of data batches in train: 423, val: 23, test: 0\t\r\nvocab size: 65\t\r\ncreating an LSTM with 2 layers\t\r\nnumber of parameters in the model: 240321\t\r\ncloning rnn\t\r\ncloning criterion\t\r\n1/21150 (epoch 0.002), train_loss =    nan, grad/param norm =    nan, time/batch = 2.80s\t\r\n2/21150 (epoch 0.005), train_loss =    nan, grad/param norm =    nan, time/batch = 2.70s\t\r\n3/21150 (epoch 0.007), train_loss =    nan, grad/param norm =    nan, time/batch = 2.72s\t\r\n4/21150 (epoch 0.009), train_loss =    nan, grad/param norm =    nan, time/batch = 2.75s\t\r\n5/21150 (epoch 0.012), train_loss =    nan, grad/param norm =    nan, time/batch = 2.71s\t\r\n6/21150 (epoch 0.014), train_loss =    nan, grad/param norm =    nan, time/batch = 2.74s\t\r\n```\r\n...and so on. This may or not be related to #28, as it results in an invalid snapshot.\r\n\r\nI've tried different models, parameters and older versions of char-rnn as well, but the error persists."},"comment":{"url":"https://api.github.com/repos/karpathy/char-rnn/issues/comments/119878550","html_url":"https://github.com/karpathy/char-rnn/issues/38#issuecomment-119878550","issue_url":"https://api.github.com/repos/karpathy/char-rnn/issues/38","id":119878550,"user":{"login":"DiscoViking","id":3620166,"avatar_url":"https://avatars.githubusercontent.com/u/3620166?v=3","gravatar_id":"","url":"https://api.github.com/users/DiscoViking","html_url":"https://github.com/DiscoViking","followers_url":"https://api.github.com/users/DiscoViking/followers","following_url":"https://api.github.com/users/DiscoViking/following{/other_user}","gists_url":"https://api.github.com/users/DiscoViking/gists{/gist_id}","starred_url":"https://api.github.com/users/DiscoViking/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DiscoViking/subscriptions","organizations_url":"https://api.github.com/users/DiscoViking/orgs","repos_url":"https://api.github.com/users/DiscoViking/repos","events_url":"https://api.github.com/users/DiscoViking/events{/privacy}","received_events_url":"https://api.github.com/users/DiscoViking/received_events","type":"User","site_admin":false},"created_at":"2015-07-09T08:50:35Z","updated_at":"2015-07-09T08:50:35Z","body":"OS = CentOS 6.5\r\nBLAS = Not sure, how do I find out?"}}