{"action":"created","issue":{"url":"https://api.github.com/repos/NervanaSystems/neon/issues/68","labels_url":"https://api.github.com/repos/NervanaSystems/neon/issues/68/labels{/name}","comments_url":"https://api.github.com/repos/NervanaSystems/neon/issues/68/comments","events_url":"https://api.github.com/repos/NervanaSystems/neon/issues/68/events","html_url":"https://github.com/NervanaSystems/neon/issues/68","id":99917676,"number":68,"title":"IOError: when writing train batches of  imagenet  ","user":{"login":"xiuxiazhang","id":6570411,"avatar_url":"https://avatars.githubusercontent.com/u/6570411?v=3","gravatar_id":"","url":"https://api.github.com/users/xiuxiazhang","html_url":"https://github.com/xiuxiazhang","followers_url":"https://api.github.com/users/xiuxiazhang/followers","following_url":"https://api.github.com/users/xiuxiazhang/following{/other_user}","gists_url":"https://api.github.com/users/xiuxiazhang/gists{/gist_id}","starred_url":"https://api.github.com/users/xiuxiazhang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiuxiazhang/subscriptions","organizations_url":"https://api.github.com/users/xiuxiazhang/orgs","repos_url":"https://api.github.com/users/xiuxiazhang/repos","events_url":"https://api.github.com/users/xiuxiazhang/events{/privacy}","received_events_url":"https://api.github.com/users/xiuxiazhang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-08-09T18:17:19Z","updated_at":"2015-08-10T17:05:12Z","closed_at":null,"body":"Hello, \r\n I am  trying to run   i1k-alexnet-fp32.yaml example, and  I have data ready in this directory \r\n\r\n[zxx@ga85 neon_dev]$ ls /mnt/bak_store01/zxx/imagenet/data/I1K/\r\nILSVRC2012_devkit_t12.tar.gz  ILSVRC2012_img_train.tar  ILSVRC2012_img_val.tar  imageset_batches\r\n\r\nand modified  two lines of examples/convnet/i1k-alexnet-fp32.yaml file\r\n\r\n47   dataset: &ds !obj:datasets.Imageset {\r\n 48     repo_path: '/mnt/bak_store01/zxx/imagenet/data/',\r\n 49     imageset: 'I1K',\r\n 50     square_crop: True,\r\n 51     save_dir: '/mnt/bak_store01/zxx/imagenet/data/I1K/imageset_batches',\r\n\r\nHowever, when I am trying to generate imageset batches, it has the IO errors. \r\nMy current neon's version is v0.8.2, since I only have kepler GPU.\r\n\r\n[zxx@ga85 neon_dev]$ neon --gpu cudanet examples/convnet/i1k-alexnet-fp32.yaml \r\n2015-08-10 02:05:01,401 INFO:batch_writer - 90% ...\r\n2015-08-10 02:05:20,666 INFO:batch_writer - created list of jpg files\r\n2015-08-10 02:05:20,666 INFO:batch_writer - Number of training files = 1281167\r\n2015-08-10 02:05:22,446 INFO:batch_writer - Writing train batches...\r\nTraceback (most recent call last):\r\n  File \"/home/zxx/Install/python2.7.10/bin/neon\", line 6, in <module>\r\n    exec(compile(open(__file__).read(), __file__, 'exec'))\r\n  File \"/home/zxx/neon_dev/bin/neon\", line 240, in <module>\r\n    experiment, result, status = main()\r\n  File \"/home/zxx/neon_dev/bin/neon\", line 208, in main\r\n    result = experiment.run()\r\n  File \"/home/zxx/neon_dev/neon/experiments/fit_predict_err.py\", line 99, in run\r\n    super(FitPredictErrorExperiment, self).run()\r\n  File \"/home/zxx/neon_dev/neon/experiments/fit.py\", line 73, in run\r\n    self.dataset.load(backend=self.backend, experiment=self)\r\n  File \"/home/zxx/neon_dev/neon/datasets/imageset.py\", line 178, in load\r\n    self.bw.run()\r\n  File \"/home/zxx/neon_dev/neon/util/batch_writer.py\", line 293, in run\r\n    t_jpegfiles, targets=None, is_tar=True)\r\n  File \"/home/zxx/neon_dev/neon/util/batch_writer.py\", line 177, in write_batches\r\n    functools.partial(proc_img, is_string=is_tar), jpeg_file_batch)\r\n  File \"/home/zxx/Install/python2.7.10/lib/python2.7/multiprocessing/pool.py\", line 251, in map\r\n    return self.map_async(func, iterable, chunksize).get()\r\n  File \"/home/zxx/Install/python2.7.10/lib/python2.7/multiprocessing/pool.py\", line 567, in get\r\n    raise self._value\r\nIOError: broken data stream when reading image file\r\n\r\nI am not familiar  with imageset IO implementation, hope someone could point out the reasons. \r\n\r\nThanks, \r\nXiuxia\r\n"},"comment":{"url":"https://api.github.com/repos/NervanaSystems/neon/issues/comments/129527807","html_url":"https://github.com/NervanaSystems/neon/issues/68#issuecomment-129527807","issue_url":"https://api.github.com/repos/NervanaSystems/neon/issues/68","id":129527807,"user":{"login":"xiuxiazhang","id":6570411,"avatar_url":"https://avatars.githubusercontent.com/u/6570411?v=3","gravatar_id":"","url":"https://api.github.com/users/xiuxiazhang","html_url":"https://github.com/xiuxiazhang","followers_url":"https://api.github.com/users/xiuxiazhang/followers","following_url":"https://api.github.com/users/xiuxiazhang/following{/other_user}","gists_url":"https://api.github.com/users/xiuxiazhang/gists{/gist_id}","starred_url":"https://api.github.com/users/xiuxiazhang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiuxiazhang/subscriptions","organizations_url":"https://api.github.com/users/xiuxiazhang/orgs","repos_url":"https://api.github.com/users/xiuxiazhang/repos","events_url":"https://api.github.com/users/xiuxiazhang/events{/privacy}","received_events_url":"https://api.github.com/users/xiuxiazhang/received_events","type":"User","site_admin":false},"created_at":"2015-08-10T17:05:12Z","updated_at":"2015-08-10T17:05:12Z","body":"I am using Redhat 6.3 OS and installed jpeg-8c.\r\n\r\n[zxx@ga85 ~]$ cat /etc/redhat-release \r\nRed Hat Enterprise Linux Server release 6.3 (Santiago)\r\n\r\n[zxx@ga85 ~]$ ldconfig -p | grep libjpeg\r\n\tlibjpeg.so.62 (libc6,x86-64) => /usr/lib64/libjpeg.so.62\r\n\t\r\nlibjpeg.so.8 (libc6,x86-64) => /usr/local/lib/libjpeg.so.8\r\n\tlibjpeg.so (libc6,x86-64) => /usr/local/lib/libjpeg.so\r\n\tlibjpeg.so (libc6,x86-64) => /usr/lib64/libjpeg.so\r\n"}}