{"action":"opened","issue":{"url":"https://api.github.com/repos/karpathy/char-rnn/issues/58","labels_url":"https://api.github.com/repos/karpathy/char-rnn/issues/58/labels{/name}","comments_url":"https://api.github.com/repos/karpathy/char-rnn/issues/58/comments","events_url":"https://api.github.com/repos/karpathy/char-rnn/issues/58/events","html_url":"https://github.com/karpathy/char-rnn/issues/58","id":93289575,"number":58,"title":"opencl and nan","user":{"login":"xonobo","id":1115812,"avatar_url":"https://avatars.githubusercontent.com/u/1115812?v=3","gravatar_id":"","url":"https://api.github.com/users/xonobo","html_url":"https://github.com/xonobo","followers_url":"https://api.github.com/users/xonobo/followers","following_url":"https://api.github.com/users/xonobo/following{/other_user}","gists_url":"https://api.github.com/users/xonobo/gists{/gist_id}","starred_url":"https://api.github.com/users/xonobo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xonobo/subscriptions","organizations_url":"https://api.github.com/users/xonobo/orgs","repos_url":"https://api.github.com/users/xonobo/repos","events_url":"https://api.github.com/users/xonobo/events{/privacy}","received_events_url":"https://api.github.com/users/xonobo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-07-06T14:14:11Z","updated_at":"2015-07-06T14:14:11Z","closed_at":null,"body":"I work on a workstation with NVIDIA card and multicore Intel CPU (opencl SDK of intel is installed).\r\n\r\nI try to use cpu as the opencl device but get nan at training and then training is aborted. I tried various options and share their outputs below. The opencl@CPU case makes compilation and vectorizations before training steps. Is this expected?\r\n\r\nI also tried training by opencl@CPU with the fork:\r\nhttps://github.com/hughperkins/char-rnn\r\nwhich goes on training with nans and floats.\r\n\r\nIs there a solution to get rid of these nans?\r\n\r\nI share lengthy outputs for the ones interested in performance comparison.\r\n\r\nAnd lastly I want to point your attention to the output of the training of fork at the last section. There are two iterations whose loss are not nan but some floating point.\r\n\r\n--------------------------------------------\r\nclTorch output:\r\n--------------------------------------------\r\nth> require('cutorch')\r\n{\r\n  streamWaitFor : function: 0x4058b558\r\n  deviceReset : function: 0x4058ba98\r\n  test : function: 0x409a58f0\r\n  _state : userdata: 0x0220bca0\r\n  streamSynchronize : function: 0x4058b928\r\n  manualSeed : function: 0x4058bf20\r\n  setStream : function: 0x4058b2b0\r\n  getMemoryUsage : function: 0x4058bcd8\r\n  setDefaultStream : function: 0x4058b488\r\n  getBlasHandle : function: 0x4058b030\r\n  CudaHostAllocator : torch.Allocator\r\n  getNumStreams : function: 0x4058b1e0\r\n  manualSeedAll : function: 0x4058bfe0\r\n  initialSeed : function: 0x4058bec0\r\n  getStream : function: 0x4058b370\r\n  setRNGState : function: 0x405840b0\r\n  setBlasHandle : function: 0x4058af68\r\n  seed : function: 0x4058bda0\r\n  getDeviceProperties : function: 0x4058bc18\r\n  reserveStreams : function: 0x4058b100\r\n  withDevice : function: 0x409a5958\r\n  setDevice : function: 0x4058bd40\r\n  seedAll : function: 0x4058be60\r\n  getNumBlasHandles : function: 0x4058af00\r\n  getDeviceCount : function: 0x4058bb58\r\n  createCudaHostTensor : function: 0x409a5998\r\n  getState : function: 0x40584170\r\n  getDevice : function: 0x4058b9b0\r\n  synchronize : function: 0x4058ad00\r\n  getRNGState : function: 0x40584048\r\n  streamWaitForMultiDevice : function: 0x4058b650\r\n  reserveBlasHandles : function: 0x4058ae30\r\n  streamBarrierMultiDevice : function: 0x4058b838\r\n  streamBarrier : function: 0x4058b720\r\n}\r\n                                                                      [1.0545s]\t\r\nth> require('cltorch')\r\n{\r\n  setAddFinish : function: 0x41f1bd90\r\n  getDeviceCount : function: 0x41f1bbd8\r\n  getDeviceProperties : function: 0x41f1bcc8\r\n  getState : function: 0x41f1bcf0\r\n  getDevice : function: 0x412c43e0\r\n  setDevice : function: 0x412c4440\r\n  _state : userdata: 0x02467900\r\n  dumpTimings : function: 0x41f1bb28\r\n  setTrace : function: 0x41f1bd40\r\n  synchronize : function: 0x412c4468\r\n  finish : function: 0x41f1bbb0\r\n}\r\n                                                                      [0.0146s]\t\r\nth> cltorch.getDeviceProperties(1)\r\n{\r\n  deviceType : \"CPU\"\r\n  maxClockFrequency : 2000\r\n  deviceName : \"       Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz\"\r\n  maxMemAllocSizeMB : 16089\r\n  globalMemCachelineSizeKB : 0\r\n  deviceVersion : \"OpenCL 1.2 (Build 43)\"\r\n  localMemSizeKB : 32\r\n  openClCVersion : \"OpenCL C 1.2 \"\r\n  maxWorkGroupSize : 8192\r\n  globalMemSizeMB : 64358\r\n  platformVendor : \"Intel(R) Corporation\"\r\n  maxComputeUnits : 32\r\n}\r\n                                                                      [0.0011s]\t\r\nth> cltorch.getDeviceProperties(2)\r\n{\r\n  deviceType : \"GPU\"\r\n  maxClockFrequency : 705\r\n  deviceName : \"Quadro 410\"\r\n  maxMemAllocSizeMB : 128\r\n  globalMemCachelineSizeKB : 0\r\n  deviceVersion : \"OpenCL 1.1 CUDA\"\r\n  localMemSizeKB : 47\r\n  openClCVersion : \"OpenCL C 1.1 \"\r\n  maxWorkGroupSize : 1024\r\n  globalMemSizeMB : 509\r\n  platformVendor : \"NVIDIA Corporation\"\r\n  maxComputeUnits : 1\r\n}\r\n\r\n--------------------------------------------\r\ntraining on CPU\r\n--------------------------------------------\r\nth train.lua -data_dir data/tinyshakespeare/ -opencl 0 -gpuid -1\r\nvocab.t7 and data.t7 do not exist. Running preprocessing...\t\r\none-time setup: preprocessing input text file data/tinyshakespeare/input.txt...\t\r\nloading text file...\t\r\ncreating vocabulary mapping...\t\r\nputting data into tensor...\t\r\nsaving data/tinyshakespeare/vocab.t7\t\r\nsaving data/tinyshakespeare/data.t7\t\r\nloading data files...\t\r\ncutting off end of data so that the batches/sequences divide evenly\t\r\nreshaping tensor...\t\r\ndata load done. Number of data batches in train: 423, val: 23, test: 0\t\r\nvocab size: 65\t\r\ncreating an LSTM with 2 layers\t\r\nnumber of parameters in the model: 240321\t\r\ncloning criterion\t\r\ncloning rnn\t\r\n1/21150 (epoch 0.002), train_loss = 4.19766416, grad/param norm = 4.5006e-01, time/batch = 1.74s\t\r\n2/21150 (epoch 0.005), train_loss = 4.10134056, grad/param norm = 6.3375e-01, time/batch = 2.14s\t\r\n3/21150 (epoch 0.007), train_loss = 3.44502399, grad/param norm = 9.4798e-01, time/batch = 1.48s\t\r\n4/21150 (epoch 0.009), train_loss = 3.45054399, grad/param norm = 1.1340e+00, time/batch = 2.03s\t\r\n5/21150 (epoch 0.012), train_loss = 3.33238818, grad/param norm = 7.8976e-01, time/batch = 1.62s\t\r\n6/21150 (epoch 0.014), train_loss = 3.37363688, grad/param norm = 7.0334e-01, time/batch = 1.61s\t\r\n7/21150 (epoch 0.017), train_loss = 3.36438210, grad/param norm = 6.5300e-01, time/batch = 2.70s\t\r\n8/21150 (epoch 0.019), train_loss = 3.33342581, grad/param norm = 7.6950e-01, time/batch = 1.72s\t\r\n9/21150 (epoch 0.021), train_loss = 3.29173263, grad/param norm = 6.1282e-01, time/batch = 1.79s\t\r\n10/21150 (epoch 0.024), train_loss = 3.38000728, grad/param norm = 4.1881e-01, time/batch = 2.92s\t\r\n\r\n--------------------------------------------\r\ntraining on GPU - CUDA\r\n--------------------------------------------\r\n th train.lua -data_dir data/tinyshakespeare/ -opencl 0 -gpuid 0\r\nusing CUDA on GPU 0...\t\r\nloading data files...\t\r\ncutting off end of data so that the batches/sequences divide evenly\t\r\nreshaping tensor...\t\r\ndata load done. Number of data batches in train: 423, val: 23, test: 0\t\r\nvocab size: 65\t\r\ncreating an LSTM with 2 layers\t\r\nnumber of parameters in the model: 240321\t\r\ncloning criterion\t\r\ncloning rnn\t\r\n1/21150 (epoch 0.002), train_loss = 4.16315975, grad/param norm = 4.5507e-01, time/batch = 0.50s\t\r\n2/21150 (epoch 0.005), train_loss = 4.06560737, grad/param norm = 6.1592e-01, time/batch = 0.45s\t\r\n3/21150 (epoch 0.007), train_loss = 3.50594769, grad/param norm = 1.2221e+00, time/batch = 0.45s\t\r\n4/21150 (epoch 0.009), train_loss = 3.45355825, grad/param norm = 1.3675e+00, time/batch = 0.44s\t\r\n5/21150 (epoch 0.012), train_loss = 3.35222242, grad/param norm = 1.2052e+00, time/batch = 0.44s\t\r\n6/21150 (epoch 0.014), train_loss = 3.37636928, grad/param norm = 8.7048e-01, time/batch = 0.44s\t\r\n7/21150 (epoch 0.017), train_loss = 3.36737326, grad/param norm = 6.1815e-01, time/batch = 0.44s\t\r\n8/21150 (epoch 0.019), train_loss = 3.32496874, grad/param norm = 4.2533e-01, time/batch = 0.44s\t\r\n9/21150 (epoch 0.021), train_loss = 3.29095509, grad/param norm = 4.5369e-01, time/batch = 0.44s\t\r\n10/21150 (epoch 0.024), train_loss = 3.38070163, grad/param norm = 4.3267e-01, time/batch = 0.44s\r\n\r\n\r\n--------------------------------------------\r\ntraining on GPU - OPENCL\r\n--------------------------------------------\r\n th train.lua -data_dir data/tinyshakespeare/ -opencl 1 -gpuid 1\r\nregistering spatialconvolutionmm\r\nusing OpenCL on GPU 1...\t\r\nloading data files...\t\r\ncutting off end of data so that the batches/sequences divide evenly\t\r\nreshaping tensor...\t\r\ndata load done. Number of data batches in train: 423, val: 23, test: 0\t\r\nvocab size: 65\t\r\ncreating an LSTM with 2 layers\t\r\nUsing NVIDIA Corporation platform: NVIDIA CUDA\r\nUsing device: Quadro 410\r\nstatefultimer v0.6\r\nnumber of parameters in the model: 240321\t\r\ncloning criterion\t\r\ncloning rnn\t\r\n1/21150 (epoch 0.002), train_loss = 4.19766393, grad/param norm = 4.5006e-01, time/batch = 1.31s\t\r\n2/21150 (epoch 0.005), train_loss = 4.10134039, grad/param norm = 6.3375e-01, time/batch = 1.10s\t\r\n3/21150 (epoch 0.007), train_loss = 3.44484827, grad/param norm = 9.4796e-01, time/batch = 1.11s\t\r\n4/21150 (epoch 0.009), train_loss = 3.45040853, grad/param norm = 1.1346e+00, time/batch = 1.10s\t\r\n5/21150 (epoch 0.012), train_loss = 3.33218116, grad/param norm = 7.8938e-01, time/batch = 1.09s\t\r\n6/21150 (epoch 0.014), train_loss = 3.37349831, grad/param norm = 7.0234e-01, time/batch = 1.04s\t\r\n7/21150 (epoch 0.017), train_loss = 3.36418301, grad/param norm = 6.5344e-01, time/batch = 0.96s\t\r\n8/21150 (epoch 0.019), train_loss = 3.33336397, grad/param norm = 7.7021e-01, time/batch = 0.95s\t\r\n9/21150 (epoch 0.021), train_loss = 3.29151368, grad/param norm = 6.1312e-01, time/batch = 0.95s\t\r\n10/21150 (epoch 0.024), train_loss = 3.37983895, grad/param norm = 4.1876e-01, time/batch = 0.95s\t\r\n\r\n--------------------------------------------\r\ntraining on CPU - OPENCL\r\n--------------------------------------------\r\n th train.lua -data_dir data/tinyshakespeare/ -opencl 1 -gpuid 0\r\nregistering spatialconvolutionmm\r\nusing OpenCL on GPU 0...\t\r\nloading data files...\t\r\ncutting off end of data so that the batches/sequences divide evenly\t\r\nreshaping tensor...\t\r\ndata load done. Number of data batches in train: 423, val: 23, test: 0\t\r\nvocab size: 65\t\r\ncreating an LSTM with 2 layers\t\r\nUsing Intel(R) Corporation platform: Intel(R) OpenCL\r\nUsing device:        Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz\r\nstatefultimer v0.6\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nnumber of parameters in the model: 240321\t\r\ncloning criterion\t\r\ncloning rnn\t\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was not vectorized\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was not vectorized\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (8)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClTensorMathTransformReduce.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_kernel_transformReduceOuterDimIndex> was successfully vectorized (4)\r\nKernel <THClTensor_kernel_transformReduceInnermostDimIndex> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was not vectorized\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClReduce.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_reduceNoncontigDim> was successfully vectorized (4)\r\nKernel <THClTensor_reduceContigDim> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was not vectorized\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\n/home/bozkalayci/torch-distro/updates/cltorch/lib/THCl/THClGather.cpp build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_kernel_gather> was successfully vectorized (4)\r\nDone.\r\nTHClReduceAll.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_reduceAll> was successfully vectorized (8)\r\nKernel <THClTensor_reduceAllPass1> was successfully vectorized (4)\r\nKernel <THClTensor_reduceAllPass2> was successfully vectorized (8)\r\nDone.\r\n/home/bozkalayci/torch-distro/updates/cltorch/lib/THCl/THClScatter.cpp build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_kernel_scatterFill> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was not vectorized\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was not vectorized\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClReduceAll.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_reduceAll> was successfully vectorized (8)\r\nKernel <THClTensor_reduceAllPass1> was successfully vectorized (4)\r\nKernel <THClTensor_reduceAllPass2> was successfully vectorized (8)\r\nDone.\r\nTHClReduceAll.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_reduceAll> was successfully vectorized (8)\r\nKernel <THClTensor_reduceAllPass1> was successfully vectorized (4)\r\nKernel <THClTensor_reduceAllPass2> was successfully vectorized (8)\r\nDone.\r\n1/21150 (epoch 0.002), train_loss =    nan, grad/param norm = 1.0629e+02, time/batch = 84.92s\t\r\nloss is exploding, aborting.\t\r\n\r\n\r\n--------------------------------------------\r\ntraining on CPU - OPENCL - fork of hughperkins\r\n--------------------------------------------\r\n th train.lua -data_dir data/tinyshakespeare/ -opencl 1 -gpuid 0\r\nregistering spatialconvolutionmm\r\nusing OpenCL on GPU 0...\t\r\nloading data files...\t\r\ncutting off end of data so that the batches/sequences divide evenly\t\r\nreshaping tensor...\t\r\ndata load done. Number of data batches in train: 423, val: 23, test: 0\t\r\nvocab size: 65\t\r\ncreating an LSTM with 2 layers\t\r\nUsing Intel(R) Corporation platform: Intel(R) OpenCL\r\nUsing device:        Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz\r\nstatefultimer v0.6\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nnumber of parameters in the model: 240321\t\r\ncloning criterion\t\r\ncloning rnn\t\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was not vectorized\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was not vectorized\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (8)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClTensorMathTransformReduce.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_kernel_transformReduceOuterDimIndex> was successfully vectorized (4)\r\nKernel <THClTensor_kernel_transformReduceInnermostDimIndex> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was not vectorized\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClReduce.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_reduceNoncontigDim> was successfully vectorized (4)\r\nKernel <THClTensor_reduceContigDim> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was not vectorized\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\n/home/bozkalayci/torch-distro/updates/cltorch/lib/THCl/THClGather.cpp build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_kernel_gather> was successfully vectorized (4)\r\nDone.\r\nTHClReduceAll.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_reduceAll> was successfully vectorized (8)\r\nKernel <THClTensor_reduceAllPass1> was successfully vectorized (4)\r\nKernel <THClTensor_reduceAllPass2> was successfully vectorized (8)\r\nDone.\r\n/home/bozkalayci/torch-distro/updates/cltorch/lib/THCl/THClScatter.cpp build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_kernel_scatterFill> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was not vectorized\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was not vectorized\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClApply.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_pointwiseApplyD> was successfully vectorized (4)\r\nDone.\r\nTHClReduceAll.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_reduceAll> was successfully vectorized (8)\r\nKernel <THClTensor_reduceAllPass1> was successfully vectorized (4)\r\nKernel <THClTensor_reduceAllPass2> was successfully vectorized (8)\r\nDone.\r\nTHClReduceAll.cl build log: \r\nCompilation started\r\nCompilation done\r\nLinking started\r\nLinking done\r\nDevice build started\r\nDevice build done\r\nKernel <THClTensor_reduceAll> was successfully vectorized (8)\r\nKernel <THClTensor_reduceAllPass1> was successfully vectorized (4)\r\nKernel <THClTensor_reduceAllPass2> was successfully vectorized (8)\r\nDone.\r\n1/21150 (epoch 0.002), train_loss =    nan, grad/param norm = 1.0629e+02, time/batch = 87.60s\t\r\n2/21150 (epoch 0.005), train_loss =    nan, grad/param norm = 1.0279e+02, time/batch = 2.42s\t\r\n3/21150 (epoch 0.007), train_loss =    nan, grad/param norm = 9.8932e+01, time/batch = 2.40s\t\r\n4/21150 (epoch 0.009), train_loss =    nan, grad/param norm = 9.5078e+01, time/batch = 2.41s\t\r\n5/21150 (epoch 0.012), train_loss =    nan, grad/param norm = 9.1377e+01, time/batch = 2.38s\t\r\n6/21150 (epoch 0.014), train_loss =    nan, grad/param norm = 8.7885e+01, time/batch = 2.43s\t\r\n7/21150 (epoch 0.017), train_loss =    nan, grad/param norm = 8.4618e+01, time/batch = 2.38s\t\r\n8/21150 (epoch 0.019), train_loss =    nan, grad/param norm = 8.1572e+01, time/batch = 2.38s\t\r\n9/21150 (epoch 0.021), train_loss =    nan, grad/param norm = 7.8735e+01, time/batch = 2.38s\t\r\n10/21150 (epoch 0.024), train_loss =    nan, grad/param norm = 7.6093e+01, time/batch = 2.38s\t\r\n11/21150 (epoch 0.026), train_loss =    nan, grad/param norm = 7.3629e+01, time/batch = 2.64s\t\r\n12/21150 (epoch 0.028), train_loss =    nan, grad/param norm = 7.1329e+01, time/batch = 2.44s\t\r\n13/21150 (epoch 0.031), train_loss =    nan, grad/param norm = 6.9178e+01, time/batch = 2.40s\t\r\n14/21150 (epoch 0.033), train_loss =    nan, grad/param norm = 6.7162e+01, time/batch = 2.40s\t\r\n15/21150 (epoch 0.035), train_loss = 4.19833310, grad/param norm = 2.8035e-01, time/batch = 2.42s\t\r\n16/21150 (epoch 0.038), train_loss =    nan, grad/param norm = 6.5226e+01, time/batch = 2.38s\t\r\n17/21150 (epoch 0.040), train_loss =    nan, grad/param norm = 6.3410e+01, time/batch = 2.39s\t\r\n18/21150 (epoch 0.043), train_loss =    nan, grad/param norm = 6.1704e+01, time/batch = 2.48s\t\r\n19/21150 (epoch 0.045), train_loss =    nan, grad/param norm = 6.0097e+01, time/batch = 2.56s\t\r\n20/21150 (epoch 0.047), train_loss =    nan, grad/param norm = 5.8581e+01, time/batch = 2.62s\t\r\n21/21150 (epoch 0.050), train_loss = 4.20001215, grad/param norm = 2.8320e-01, time/batch = 2.57s\t\r\n22/21150 (epoch 0.052), train_loss =    nan, grad/param norm = 5.7113e+01, time/batch = 2.43s\t\r\n23/21150 (epoch 0.054), train_loss =    nan, grad/param norm = 5.5728e+01, time/batch = 2.40s\t\r\n24/21150 (epoch 0.057), train_loss =    nan, grad/param norm = 5.4416e+01, time/batch = 2.38s\t\r\n25/21150 (epoch 0.059), train_loss =    nan, grad/param norm = 5.3173e+01, time/batch = 2.37s\t\r\n\r\n"}}