{"action":"created","issue":{"url":"https://api.github.com/repos/BVLC/caffe/issues/1479","labels_url":"https://api.github.com/repos/BVLC/caffe/issues/1479/labels{/name}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/1479/comments","events_url":"https://api.github.com/repos/BVLC/caffe/issues/1479/events","html_url":"https://github.com/BVLC/caffe/pull/1479","id":50143841,"number":1479,"title":"Added CHECK if loss is NaN","user":{"login":"sguada","id":1766524,"avatar_url":"https://avatars.githubusercontent.com/u/1766524?v=3","gravatar_id":"","url":"https://api.github.com/users/sguada","html_url":"https://github.com/sguada","followers_url":"https://api.github.com/users/sguada/followers","following_url":"https://api.github.com/users/sguada/following{/other_user}","gists_url":"https://api.github.com/users/sguada/gists{/gist_id}","starred_url":"https://api.github.com/users/sguada/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sguada/subscriptions","organizations_url":"https://api.github.com/users/sguada/orgs","repos_url":"https://api.github.com/users/sguada/repos","events_url":"https://api.github.com/users/sguada/events{/privacy}","received_events_url":"https://api.github.com/users/sguada/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/BVLC/caffe/labels/JL","name":"JL","color":"fef2c0"},{"url":"https://api.github.com/repos/BVLC/caffe/labels/needs%20rebase","name":"needs rebase","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2014-11-26T10:48:34Z","updated_at":"2015-07-09T17:23:18Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/BVLC/caffe/pulls/1479","html_url":"https://github.com/BVLC/caffe/pull/1479","diff_url":"https://github.com/BVLC/caffe/pull/1479.diff","patch_url":"https://github.com/BVLC/caffe/pull/1479.patch"},"body":"As mentioned in #1349 If Loss is Nan it doesn't make sense to keep training."},"comment":{"url":"https://api.github.com/repos/BVLC/caffe/issues/comments/120077931","html_url":"https://github.com/BVLC/caffe/pull/1479#issuecomment-120077931","issue_url":"https://api.github.com/repos/BVLC/caffe/issues/1479","id":120077931,"user":{"login":"PiranjaF","id":9516330,"avatar_url":"https://avatars.githubusercontent.com/u/9516330?v=3","gravatar_id":"","url":"https://api.github.com/users/PiranjaF","html_url":"https://github.com/PiranjaF","followers_url":"https://api.github.com/users/PiranjaF/followers","following_url":"https://api.github.com/users/PiranjaF/following{/other_user}","gists_url":"https://api.github.com/users/PiranjaF/gists{/gist_id}","starred_url":"https://api.github.com/users/PiranjaF/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PiranjaF/subscriptions","organizations_url":"https://api.github.com/users/PiranjaF/orgs","repos_url":"https://api.github.com/users/PiranjaF/repos","events_url":"https://api.github.com/users/PiranjaF/events{/privacy}","received_events_url":"https://api.github.com/users/PiranjaF/received_events","type":"User","site_admin":false},"created_at":"2015-07-09T17:23:17Z","updated_at":"2015-07-09T17:23:17Z","body":"How would I go about stopping training more silently? With this implementation any script in pycaffe would still crash immediately. It should be possible to catch the error or perhaps set as an option that training stops without stopping everything else."}}