{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/cxxnet/issues/241","labels_url":"https://api.github.com/repos/dmlc/cxxnet/issues/241/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/cxxnet/issues/241/comments","events_url":"https://api.github.com/repos/dmlc/cxxnet/issues/241/events","html_url":"https://github.com/dmlc/cxxnet/issues/241","id":108248178,"number":241,"title":"problems when training ImageNet","user":{"login":"zjubxl123","id":13829403,"avatar_url":"https://avatars.githubusercontent.com/u/13829403?v=3","gravatar_id":"","url":"https://api.github.com/users/zjubxl123","html_url":"https://github.com/zjubxl123","followers_url":"https://api.github.com/users/zjubxl123/followers","following_url":"https://api.github.com/users/zjubxl123/following{/other_user}","gists_url":"https://api.github.com/users/zjubxl123/gists{/gist_id}","starred_url":"https://api.github.com/users/zjubxl123/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zjubxl123/subscriptions","organizations_url":"https://api.github.com/users/zjubxl123/orgs","repos_url":"https://api.github.com/users/zjubxl123/repos","events_url":"https://api.github.com/users/zjubxl123/events{/privacy}","received_events_url":"https://api.github.com/users/zjubxl123/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-09-25T02:07:48Z","updated_at":"2015-09-25T02:20:07Z","closed_at":null,"body":"I installed cxxnet successfully and it can run on Mnist dataset. When I tried cxxnet on ImageNet Dataset, it can still run, but it may halt unexpectedly and randomly, sometimes when loading the mean_224.bin, sometimes when training some batches of images, as shown below. And no error prompted, just stayed with no changes in number of batches and time elapsed, and the same status lasts more than several hours. I can only kill the process with Ctrl+C. \r\nThe memory of the computer is enough with 5GB used by cxxnet and 2GB left free.\r\n![default](https://cloud.githubusercontent.com/assets/13829403/10091040/b7968970-6368-11e5-902f-7588e6386e6e.PNG)\r\n\r\n\r\nI used kaiming.conf as shown below when running cxxnet. \r\n# Configuration for ImageNet  \r\n# Acknowledgement:\r\n#  Ref: He, Kaiming, and Jian Sun. \"Convolutional Neural Networks at Constrained Time Cost.\" CVPR2015\r\n# J' model in the paper above\r\n\r\ndata = train\r\niter = imgrec\r\n\r\n  image_rec  = \"train.bin\"\r\n  image_mean = \"mean_224.bin\"\r\n  rand_crop=1\r\n  rand_mirror=1\r\n  min_crop_size=192\r\n  max_crop_size=224\r\n  max_aspect_ratio=0.3\r\niter = threadbuffer\r\niter = end\r\n\r\neval = test\r\niter = imgrec\r\n\r\n  image_rec = \"test.bin\"\r\n  image_mean = \"mean_224.bin\"\r\n\r\niter = end\r\n\r\n###### Stage 1 #######\r\nnetconfig=start\r\nlayer[0->1] = conv:conv1\r\n  kernel_size = 7\r\n  stride = 2\r\n  nchannel = 64\r\nlayer[1->2] = relu:relu1\r\nlayer[2->3] = max_pooling\r\n  kernel_size = 3\r\n\r\n###### Stage 2 #######\r\nlayer[3->4] = conv:conv2\r\n  nchannel = 128\r\n  kernel_size = 2\r\n  stride = 3\r\nlayer[4->5] = relu:relu2\r\n\r\nlayer[5->6] = conv:conv3\r\n  nchannel = 128\r\n  kernel_size = 2\r\n  pad = 1\r\nlayer[6->7] = relu:relu3\r\n\r\nlayer[7->8] = conv:conv4\r\n  nchannel = 128\r\n  kernel_size = 2\r\nlayer[8->9] = relu:relu4\r\n\r\nlayer[9->10] = conv:conv5\r\n  nchannel = 128\r\n  kernel_size = 2\r\n  pad = 1\r\nlayer[10->11] = relu:relu5\r\n\r\nlayer[11->12] = max_pooling:pool1\r\n  kernel_size = 3\r\n\r\n###### Stage 3 #######\r\nlayer[12->13] = conv:conv6\r\n  nchannel = 256\r\n  kernel_size = 2\r\n  stride = 2\r\nlayer[13->14] = relu:relu6\r\n\r\nlayer[14->15] = conv:conv7\r\n  nchannel = 256\r\n  kernel_size = 2\r\n  pad = 1\r\nlayer[15->16] = relu:relu7\r\n\r\nlayer[16->17] = conv:conv8\r\n  nchannel = 256\r\n  kernel_size = 2\r\nlayer[17->18] = relu:relu8\r\n\r\nlayer[18->19] = conv:conv9\r\n  nchannel = 256\r\n  kernel_size = 2\r\n  pad = 1\r\nlayer[19->20] = relu:relu9\r\n\r\nlayer[20->21] = max_pooling:pool2\r\n  kernel_size = 3\r\n\r\n###### Stage 4 #######\r\nlayer[21->22] = conv:conv10\r\n  nchannel = 2304\r\n  kernel_size = 2\r\n  stride = 3\r\nlayer[22->23] = relu:relu10\r\n\r\nlayer[23->24] = conv:conv11\r\n  nchannel = 256\r\n  kernel_size = 2\r\n  pad = 1\r\nlayer[24->25] = relu:relu11\r\n\r\n###### Stage 5 #######\r\nlayer[25->26,27,28,29] = split:split1\r\nlayer[26->30] = max_pooling:pool3\r\n  kernel_size = 1\r\n  stride = 1\r\nlayer[27->31] = max_pooling:pool4\r\n  kernel_size = 2\r\n  stride = 2\r\nlayer[28->32] = max_pooling:pool5\r\n  kernel_size = 3\r\n  stride = 3\r\nlayer[29->33] = max_pooling:pool6\r\n  kernel_size = 6\r\n  stride = 6\r\n\r\nlayer[30->34] = flatten:f1\r\nlayer[31->35] = flatten:f2\r\nlayer[32->36] = flatten:f3\r\nlayer[33->37] = flatten:f4\r\nlayer[34,35,36,37->38] = concat:concat1\r\n\r\n###### Stage 6 #######\r\nlayer[38->39] = fullc:fc1\r\n  nhidden = 4096\r\nlayer[39->40] = relu:relu12\r\nlayer[40->40] = dropout\r\n  threshold = 0.5\r\n\r\nlayer[40->41] = fullc:fc2\r\n  nhidden = 4096\r\nlayer[41->42] = relu:relu13\r\nlayer[42->42] = dropout\r\n  threshold = 0.5\r\n\r\nlayer[42->43] = fullc:fc3\r\n  nhidden = 1000\r\nlayer[43->43] = softmax:softmax1\r\nnetconfig=end\r\n\r\n# evaluation metric\r\nmetric = rec@1\r\nmetric = rec@5\r\n\r\nmax_round = 100\r\nnum_round = 100\r\n\r\n# input shape not including batch\r\ninput_shape = 3,224,224\r\n\r\nbatch_size = 256\r\n\r\n# global parameters in any sectiion outside netconfig, and iter\r\nmomentum = 0.9\r\nwmat:lr  = 0.01\r\nwmat:wd  = 0.0005\r\n\r\nbias:wd  = 0.000\r\nbias:lr  = 0.02\r\n\r\n# all the learning rate schedule starts with lr\r\nlr:schedule = factor\r\nlr:gamma = 0.1\r\nlr:step = 300000\r\n\r\nsave_model=1\r\nmodel_dir=models\r\nprint_step=1\r\n\r\n# random config\r\nrandom_type = xavier\r\n\r\n# new line\r\ndev = gpu:0,1\r\n#dev = cpu\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/dmlc/cxxnet/issues/comments/143105580","html_url":"https://github.com/dmlc/cxxnet/issues/241#issuecomment-143105580","issue_url":"https://api.github.com/repos/dmlc/cxxnet/issues/241","id":143105580,"user":{"login":"antinucleon","id":1258974,"avatar_url":"https://avatars.githubusercontent.com/u/1258974?v=3","gravatar_id":"","url":"https://api.github.com/users/antinucleon","html_url":"https://github.com/antinucleon","followers_url":"https://api.github.com/users/antinucleon/followers","following_url":"https://api.github.com/users/antinucleon/following{/other_user}","gists_url":"https://api.github.com/users/antinucleon/gists{/gist_id}","starred_url":"https://api.github.com/users/antinucleon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antinucleon/subscriptions","organizations_url":"https://api.github.com/users/antinucleon/orgs","repos_url":"https://api.github.com/users/antinucleon/repos","events_url":"https://api.github.com/users/antinucleon/events{/privacy}","received_events_url":"https://api.github.com/users/antinucleon/received_events","type":"User","site_admin":false},"created_at":"2015-09-25T02:20:07Z","updated_at":"2015-09-25T02:20:07Z","body":"Sorry we are going to deprecate cxxnet project and fully transfer to [MXNET](https://github.com/dmlc/mxnet) in next a few days (or next week). We are still working on some final doc and final debug/feature. If you like you make take a look of https://github.com/dmlc/mxnet/blob/master/example/imagenet/README.md"}}