{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/942","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/942/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/942/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/942/events","html_url":"https://github.com/dmlc/mxnet/issues/942","id":122272991,"number":942,"title":"Get nan while training","user":{"login":"were","id":2356965,"avatar_url":"https://avatars.githubusercontent.com/u/2356965?v=3","gravatar_id":"","url":"https://api.github.com/users/were","html_url":"https://github.com/were","followers_url":"https://api.github.com/users/were/followers","following_url":"https://api.github.com/users/were/following{/other_user}","gists_url":"https://api.github.com/users/were/gists{/gist_id}","starred_url":"https://api.github.com/users/were/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/were/subscriptions","organizations_url":"https://api.github.com/users/were/orgs","repos_url":"https://api.github.com/users/were/repos","events_url":"https://api.github.com/users/were/events{/privacy}","received_events_url":"https://api.github.com/users/were/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-12-15T13:30:04Z","updated_at":"2015-12-15T17:39:42Z","closed_at":null,"body":"I wrote a mxnet-code to build MLP, but I get `nan` while training...\r\nMy code works well on other trainings, but failed on this.\r\n\r\n`````\r\nimport mxnet as mx\r\nimport numpy\r\nimport sys\r\nimport logging\r\nimport math\r\nimport random\r\n\r\nlogging.basicConfig(level = logging.DEBUG)\r\n\r\nclass MyMetric(mx.metric.EvalMetric):\r\n    def __init__(self):\r\n        self.name = 'MSE'\r\n        self.reset()\r\n\r\n    def reset(self):\r\n        self.res = 0.\r\n\r\n    def eval_it(self, label, pred):\r\n        pred = pred.asnumpy()\r\n        label = label.asnumpy()\r\n        res = 0\r\n        #cnt = 0\r\n        for i, j in zip(label, pred):\r\n            delta = ((j - i) * (j - i)).sum() / i.shape[0]\r\n            res += math.sqrt(delta)\r\n            #cnt += 1 print cnt, i, j\r\n        return res\r\n\r\n    def update(self, label, pred):\r\n        assert len(label) == len(pred)\r\n        for i, j in zip(label, pred):\r\n            self.res += self.eval_it(i, j)\r\n\r\n    def get(self):\r\n        return (self.name, self.res)\r\n\r\nargs = sys.argv[1:]\r\ntry:\r\n    print args\r\n    n_layers = int(args[0])\r\nexcept:\r\n    print 'The first argument should be number of layers!'\r\n    quit()\r\n\r\nnn = None\r\ntry:\r\n    nn = mx.symbol.Variable('data')\r\n    print 'shape is', nn.infer_shape\r\nexcept:\r\n    print 'Failed to initialize input layer!'\r\n    quit()\r\n\r\ntry:\r\n    for i in range(2, n_layers):\r\n        nn = mx.symbol.FullyConnected(data = nn, name = 'hidden-recept-' + str(i), num_hidden = int(args[i]))\r\n        nn = mx.symbol.Activation(data = nn, name = 'hidden-act-' + str(i), act_type = 'softrelu')\r\n        #if random.randint(0, 1):\r\n            #nn = mx.symbol.Dropout(data = nn, name = 'hidden-dropout', p = random.random())\r\nexcept:\r\n    print 'Failed to initailize hidden layer!'\r\n    quit()\r\n\r\ntry:\r\n    nn = mx.symbol.FullyConnected(data = nn, name = 'hidden-recept-' + str(n_layers), num_hidden = int(args[n_layers]),\r\n            no_bias = True)\r\n    nn = mx.symbol.LinearRegressionOutput(data = nn, name = 'softmax')\r\nexcept:\r\n    print 'Failed to build output layer!'\r\n    quit()\r\n\r\nBATCH_SIZE = 2048\r\n\r\ndata = open('data').readlines()\r\nx,y = [],[]\r\nfor i in xrange(len(data)):\r\n    if i % 2 == 0:\r\n        x.append(map(float, data[i].split()))\r\n    else:\r\n        y.append(map(float, data[i].split()))\r\nx = mx.ndarray.array(x).asnumpy()\r\ny = mx.ndarray.array(y).asnumpy()\r\ndata_iter = mx.io.NDArrayIter(data = x, label = y, batch_size = BATCH_SIZE)\r\n\r\n#print iterator.provide_data, iterator.provide_label\r\n\r\ndata = open('test').readlines()\r\nx,y = [],[]\r\nmagnitude = 0.\r\nfor i in xrange(len(data)):\r\n    tmp = map(float, data[i].split())\r\n    if i % 2 == 0:\r\n        x.append(tmp)\r\n    else:\r\n        y.append(tmp)\r\nx = mx.ndarray.array(x).asnumpy()\r\ny = mx.ndarray.array(y).asnumpy()\r\ntest_iter = mx.io.NDArrayIter(data = x, label = y, batch_size = BATCH_SIZE)\r\n#print provide_data, provide_label\r\n\r\nmon = mx.monitor.Monitor(1)\r\n#mon = None\r\n\r\n#mx.visualization.plot_network(nn)\r\n\r\nmodel = mx.model.FeedForward(\r\n        symbol = nn,\r\n        ctx = mx.gpu(0),\r\n        num_epoch = 1000,\r\n        optimizer = mx.optimizer.RMSProp(learning_rate = 1e-2),\r\n        initializer = mx.initializer.Xavier(magnitude = .3)\r\n)\r\n\r\nmodel.fit(\r\n        X = data_iter,\r\n        monitor = mon,\r\n        eval_data = test_iter,\r\n        eval_metric = MyMetric(),\r\n        batch_end_callback = mx.callback.Speedometer(BATCH_SIZE, BATCH_SIZE)\r\n)\r\n\r\nmodel.save(\"res\")\r\n\r\ntest_iter.reset()\r\nfor i in model.predict(test_iter):\r\n    print i\r\n`````\r\n\r\nuse `python builder.py 5 2 4 4 4 2` to build a 5-layer MLP, which 4 neurons in each hidden layer.\r\n`data` and `test` are files of training and testing data.\r\nboth `test_iter` and `data_iter` are iterator for two `2*n NDArray`.\r\n\r\nI tried to add a monitor to see what happened.\r\nIt told me the training process runs well on first 4 batches, and starts to be `nan` since 5.\r\n\r\n``````\r\nINFO:root:Batch:       1 hidden-recept-2_output         0.165509\r\nINFO:root:Batch:       1 hidden-act-2_output            0.773078\r\nINFO:root:Batch:       1 hidden-recept-3_output         0.145818\r\nINFO:root:Batch:       1 hidden-act-3_output            0.722256\r\nINFO:root:Batch:       1 hidden-recept-4_output         0.280935\r\nINFO:root:Batch:       1 hidden-act-4_output            0.832174\r\nINFO:root:Batch:       1 hidden-recept-5_output         0.427171\r\nINFO:root:Batch:       1 softmax_output                 0.427171\r\nINFO:root:Batch:       1 softmax_backward_data          0.518878\r\nINFO:root:Batch:       1 softmax_backward_label         inf\r\nINFO:root:Batch:       1 hidden-recept-5_backward_data  0.132449\r\nINFO:root:Batch:       1 hidden-recept-5_backward_weight 795.523\r\nINFO:root:Batch:       1 hidden-act-4_backward_data     0.0738911\r\nINFO:root:Batch:       1 hidden-recept-4_backward_data  0.0274807\r\nINFO:root:Batch:       1 hidden-recept-4_backward_weight 96.4226\r\nINFO:root:Batch:       1 hidden-recept-4_backward_bias  133.524\r\nINFO:root:Batch:       1 hidden-act-3_backward_data     0.0146831\r\nINFO:root:Batch:       1 hidden-recept-3_backward_data  0.00441014\r\nINFO:root:Batch:       1 hidden-recept-3_backward_weight 20.4973\r\nINFO:root:Batch:       1 hidden-recept-3_backward_bias  26.6551\r\nINFO:root:Batch:       1 hidden-act-2_backward_data     0.00228981\r\nINFO:root:Batch:       1 hidden-recept-2_backward_data  0.518878\r\nINFO:root:Batch:       1 hidden-recept-2_backward_weight 2.26191\r\nINFO:root:Batch:       1 hidden-recept-2_backward_bias  4.22354\r\nINFO:root:Batch:       1 data                           0.637145\r\nINFO:root:Batch:       1 hidden-recept-2_weight         0.139162\r\nINFO:root:Batch:       1 hidden-recept-2_bias           0.0458743\r\nINFO:root:Batch:       1 hidden-recept-3_weight         0.125683\r\nINFO:root:Batch:       1 hidden-recept-3_bias           0.0458821\r\nINFO:root:Batch:       1 hidden-recept-4_weight         0.164311\r\nINFO:root:Batch:       1 hidden-recept-4_bias           0.0458831\r\nINFO:root:Batch:       1 hidden-recept-5_weight         0.191803\r\nINFO:root:Batch:       1 softmax_label                  0.911673\r\nINFO:root:Batch:       2 hidden-recept-2_output         0.158495\r\nINFO:root:Batch:       2 hidden-act-2_output            0.75144\r\nINFO:root:Batch:       2 hidden-recept-3_output         0.120121\r\nINFO:root:Batch:       2 hidden-act-3_output            0.6759\r\nINFO:root:Batch:       2 hidden-recept-4_output         0.200911\r\nINFO:root:Batch:       2 hidden-act-4_output            0.764085\r\nINFO:root:Batch:       2 hidden-recept-5_output         0.403537\r\nINFO:root:Batch:       2 softmax_output                 0.403537\r\nINFO:root:Batch:       2 softmax_backward_data          0.438727\r\nINFO:root:Batch:       2 softmax_backward_label         0.0743718\r\nINFO:root:Batch:       2 hidden-recept-5_backward_data  0.103892\r\nINFO:root:Batch:       2 hidden-recept-5_backward_weight 587.519\r\nINFO:root:Batch:       2 hidden-act-4_backward_data     0.0559733\r\nINFO:root:Batch:       2 hidden-recept-4_backward_data  0.0141433\r\nINFO:root:Batch:       2 hidden-recept-4_backward_weight 61.7605\r\nINFO:root:Batch:       2 hidden-recept-4_backward_bias  91.3926\r\nINFO:root:Batch:       2 hidden-act-3_backward_data     0.00706452\r\nINFO:root:Batch:       2 hidden-recept-3_backward_data  0.00205192\r\nINFO:root:Batch:       2 hidden-recept-3_backward_weight 7.46835\r\nINFO:root:Batch:       2 hidden-recept-3_backward_bias  9.99114\r\nINFO:root:Batch:       2 hidden-act-2_backward_data     0.00110245\r\nINFO:root:Batch:       2 hidden-recept-2_backward_data  0.438727\r\nINFO:root:Batch:       2 hidden-recept-2_backward_weight 0.979812\r\nINFO:root:Batch:       2 hidden-recept-2_backward_bias  1.78149\r\nINFO:root:Batch:       2 data                           0.639046\r\nINFO:root:Batch:       2 hidden-recept-2_weight         0.156327\r\nINFO:root:Batch:       2 hidden-recept-2_bias           0.11099\r\nINFO:root:Batch:       2 hidden-recept-3_weight         0.135165\r\nINFO:root:Batch:       2 hidden-recept-3_bias           0.0927643\r\nINFO:root:Batch:       2 hidden-recept-4_weight         0.17869\r\nINFO:root:Batch:       2 hidden-recept-4_bias           0.11218\r\nINFO:root:Batch:       2 hidden-recept-5_weight         0.215187\r\nINFO:root:Batch:       2 softmax_label                  0.909899\r\nINFO:root:Batch:       3 hidden-recept-2_output         0.236518\r\nINFO:root:Batch:       3 hidden-act-2_output            0.737678\r\nINFO:root:Batch:       3 hidden-recept-3_output         0.28877\r\nINFO:root:Batch:       3 hidden-act-3_output            0.693304\r\nINFO:root:Batch:       3 hidden-recept-4_output         0.375687\r\nINFO:root:Batch:       3 hidden-act-4_output            0.726632\r\nINFO:root:Batch:       3 hidden-recept-5_output         0.494087\r\nINFO:root:Batch:       3 softmax_output                 0.494087\r\nINFO:root:Batch:       3 softmax_backward_data          0.349887\r\nINFO:root:Batch:       3 softmax_backward_label         0.0550792\r\nINFO:root:Batch:       3 hidden-recept-5_backward_data  0.0891329\r\nINFO:root:Batch:       3 hidden-recept-5_backward_weight 396.782\r\nINFO:root:Batch:       3 hidden-act-4_backward_data     0.0499654\r\nINFO:root:Batch:       3 hidden-recept-4_backward_data  0.0180972\r\nINFO:root:Batch:       3 hidden-recept-4_backward_weight 46.9512\r\nINFO:root:Batch:       3 hidden-recept-4_backward_bias  67.844\r\nINFO:root:Batch:       3 hidden-act-3_backward_data     0.00866769\r\nINFO:root:Batch:       3 hidden-recept-3_backward_data  0.00255783\r\nINFO:root:Batch:       3 hidden-recept-3_backward_weight 9.1302\r\nINFO:root:Batch:       3 hidden-recept-3_backward_bias  12.3923\r\nINFO:root:Batch:       3 hidden-act-2_backward_data     0.00135789\r\nINFO:root:Batch:       3 hidden-recept-2_backward_data  0.349887\r\nINFO:root:Batch:       3 hidden-recept-2_backward_weight 1.08412\r\nINFO:root:Batch:       3 hidden-recept-2_backward_bias  1.87598\r\nINFO:root:Batch:       3 data                           0.637617\r\nINFO:root:Batch:       3 hidden-recept-2_weight         0.20213\r\nINFO:root:Batch:       3 hidden-recept-2_bias           0.188711\r\nINFO:root:Batch:       3 hidden-recept-3_weight         0.160229\r\nINFO:root:Batch:       3 hidden-recept-3_bias           0.139786\r\nINFO:root:Batch:       3 hidden-recept-4_weight         0.2199\r\nINFO:root:Batch:       3 hidden-recept-4_bias           0.185218\r\nINFO:root:Batch:       3 hidden-recept-5_weight         0.257822\r\nINFO:root:Batch:       3 softmax_label                  0.90779\r\nINFO:root:Batch:       4 hidden-recept-2_output         0.373694\r\nINFO:root:Batch:       4 hidden-act-2_output            0.74138\r\nINFO:root:Batch:       4 hidden-recept-3_output         0.495543\r\nINFO:root:Batch:       4 hidden-act-3_output            0.790558\r\nINFO:root:Batch:       4 hidden-recept-4_output         0.705683\r\nINFO:root:Batch:       4 hidden-act-4_output            0.770608\r\nINFO:root:Batch:       4 hidden-recept-5_output         0.688595\r\nINFO:root:Batch:       4 softmax_output                 0.688595\r\nINFO:root:Batch:       4 softmax_backward_data          0.270061\r\nINFO:root:Batch:       4 softmax_backward_label         0.0496056\r\nINFO:root:Batch:       4 hidden-recept-5_backward_data  0.0886355\r\nINFO:root:Batch:       4 hidden-recept-5_backward_weight 232.365\r\nINFO:root:Batch:       4 hidden-act-4_backward_data     0.0506027\r\nINFO:root:Batch:       4 hidden-recept-4_backward_data  0.0245503\r\nINFO:root:Batch:       4 hidden-recept-4_backward_weight 25.6307\r\nINFO:root:Batch:       4 hidden-recept-4_backward_bias  32.7064\r\nINFO:root:Batch:       4 hidden-act-3_backward_data     0.0113317\r\nINFO:root:Batch:       4 hidden-recept-3_backward_data  0.00345556\r\nINFO:root:Batch:       4 hidden-recept-3_backward_weight 7.18837\r\nINFO:root:Batch:       4 hidden-recept-3_backward_bias  9.85327\r\nINFO:root:Batch:       4 hidden-act-2_backward_data     0.00181982\r\nINFO:root:Batch:       4 hidden-recept-2_backward_data  0.270061\r\nINFO:root:Batch:       4 hidden-recept-2_backward_weight 1.21468\r\nINFO:root:Batch:       4 hidden-recept-2_backward_bias  1.97961\r\nINFO:root:Batch:       4 data                           0.641723\r\nINFO:root:Batch:       4 hidden-recept-2_weight         0.267277\r\nINFO:root:Batch:       4 hidden-recept-2_bias           0.272548\r\nINFO:root:Batch:       4 hidden-recept-3_weight         0.201508\r\nINFO:root:Batch:       4 hidden-recept-3_bias           0.194353\r\nINFO:root:Batch:       4 hidden-recept-4_weight         0.274731\r\nINFO:root:Batch:       4 hidden-recept-4_bias           0.258076\r\nINFO:root:Batch:       4 hidden-recept-5_weight         0.303316\r\nINFO:root:Batch:       4 softmax_label                  0.892841\r\nINFO:root:Batch:       5 hidden-recept-2_output         0.534661\r\nINFO:root:Batch:       5 hidden-act-2_output            0.777116\r\nINFO:root:Batch:       5 hidden-recept-3_output         0.749784\r\nINFO:root:Batch:       5 hidden-act-3_output            0.957261\r\nINFO:root:Batch:       5 hidden-recept-4_output         1.13733\r\nINFO:root:Batch:       5 hidden-act-4_output            0.920114\r\nINFO:root:Batch:       5 hidden-recept-5_output         0.946593\r\nINFO:root:Batch:       5 softmax_output                 0.946593\r\nINFO:root:Batch:       5 softmax_backward_data          nan\r\nINFO:root:Batch:       5 softmax_backward_label         0.0500394\r\nINFO:root:Batch:       5 hidden-recept-5_backward_data  nan\r\nINFO:root:Batch:       5 hidden-recept-5_backward_weight nan\r\nINFO:root:Batch:       5 hidden-act-4_backward_data     nan\r\nINFO:root:Batch:       5 hidden-recept-4_backward_data  nan\r\nINFO:root:Batch:       5 hidden-recept-4_backward_weight nan\r\nINFO:root:Batch:       5 hidden-recept-4_backward_bias  nan\r\nINFO:root:Batch:       5 hidden-act-3_backward_data     nan\r\nINFO:root:Batch:       5 hidden-recept-3_backward_data  nan\r\nINFO:root:Batch:       5 hidden-recept-3_backward_weight nan\r\nINFO:root:Batch:       5 hidden-recept-3_backward_bias  nan\r\nINFO:root:Batch:       5 hidden-act-2_backward_data     nan\r\nINFO:root:Batch:       5 hidden-recept-2_backward_data  nan\r\nINFO:root:Batch:       5 hidden-recept-2_backward_weight nan\r\nINFO:root:Batch:       5 hidden-recept-2_backward_bias  nan\r\nINFO:root:Batch:       5 data                           0.640956\r\nINFO:root:Batch:       5 hidden-recept-2_weight         nan\r\nINFO:root:Batch:       5 hidden-recept-2_bias           nan\r\nINFO:root:Batch:       5 hidden-recept-3_weight         nan\r\nINFO:root:Batch:       5 hidden-recept-3_bias           nan\r\nINFO:root:Batch:       5 hidden-recept-4_weight         nan\r\nINFO:root:Batch:       5 hidden-recept-4_bias           nan\r\nINFO:root:Batch:       5 hidden-recept-5_weight         nan\r\nINFO:root:Batch:       5 softmax_label                  nan\r\nINFO:root:Batch:       6 hidden-recept-2_output         nan\r\n``````"},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/164837355","html_url":"https://github.com/dmlc/mxnet/issues/942#issuecomment-164837355","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/942","id":164837355,"user":{"login":"were","id":2356965,"avatar_url":"https://avatars.githubusercontent.com/u/2356965?v=3","gravatar_id":"","url":"https://api.github.com/users/were","html_url":"https://github.com/were","followers_url":"https://api.github.com/users/were/followers","following_url":"https://api.github.com/users/were/following{/other_user}","gists_url":"https://api.github.com/users/were/gists{/gist_id}","starred_url":"https://api.github.com/users/were/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/were/subscriptions","organizations_url":"https://api.github.com/users/were/orgs","repos_url":"https://api.github.com/users/were/repos","events_url":"https://api.github.com/users/were/events{/privacy}","received_events_url":"https://api.github.com/users/were/received_events","type":"User","site_admin":false},"created_at":"2015-12-15T17:39:42Z","updated_at":"2015-12-15T17:39:42Z","body":"Close the issue."}}