{"action":"closed","issue":{"url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/374","labels_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/374/labels{/name}","comments_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/374/comments","events_url":"https://api.github.com/repos/kaldi-asr/kaldi/issues/374/events","html_url":"https://github.com/kaldi-asr/kaldi/issues/374","id":119823314,"number":374,"title":"Segfault in cublasDestroy_v2 in nnet/nnet-randomizer-test","user":{"login":"kkm000","id":8228229,"avatar_url":"https://avatars.githubusercontent.com/u/8228229?v=3","gravatar_id":"","url":"https://api.github.com/users/kkm000","html_url":"https://github.com/kkm000","followers_url":"https://api.github.com/users/kkm000/followers","following_url":"https://api.github.com/users/kkm000/following{/other_user}","gists_url":"https://api.github.com/users/kkm000/gists{/gist_id}","starred_url":"https://api.github.com/users/kkm000/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kkm000/subscriptions","organizations_url":"https://api.github.com/users/kkm000/orgs","repos_url":"https://api.github.com/users/kkm000/repos","events_url":"https://api.github.com/users/kkm000/events{/privacy}","received_events_url":"https://api.github.com/users/kkm000/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-12-01T22:31:13Z","updated_at":"2015-12-10T00:57:49Z","closed_at":"2015-12-10T00:57:49Z","body":"The test crashes upon unwinding, after outputting \"Tests succeeded.\"\r\n\r\nI did not upgrade anything essential on the machine for a while, and certain that this test used to pass with flying colors. I did not however update the files I am working with possibly for a few months.\r\n\r\nI can try to bisect a revision range but if @danpovey or @jtrmal or anyone had an idea of a recent change that could have caused this.\r\n\r\nToolchain versions and valgrind memcheck attached.\r\n```\r\nkkm@yupana:~/work/kaldi/src[work]$ gcc --version\r\ngcc (Ubuntu 4.9.3-5ubuntu1~14.04) 4.9.3\r\n\r\nkkm@yupana:~/work/kaldi/src[work]$ ldd --version\r\nldd (Ubuntu EGLIBC 2.19-0ubuntu6.6) 2.19\r\n\r\nkkm@yupana:~/work/kaldi/src[work]$ nvcc --version\r\nCuda compilation tools, release 7.0, V7.0.27\r\n\r\nkkm@yupana:~/work/kaldi/src[work]$ valgrind nnet/nnet-randomizer-test\r\n==7839== Memcheck, a memory error detector\r\n==7839== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.\r\n==7839== Using Valgrind-3.10.0.SVN and LibVEX; rerun with -h for copyright info\r\n==7839== Command: nnet/nnet-randomizer-test\r\n==7839==\r\nLOG (Init():nnet-randomizer.cc:31) Seeding by srand with : 777\r\n==7839== Warning: noted but unhandled ioctl 0x30000001 with no size/direction hints\r\n==7839==    This could cause spurious value errors to appear.\r\n==7839==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.\r\n==7839== Warning: set address range perms: large range [0x1000000000, 0x1900000000) (noaccess)\r\n==7839== Warning: set address range perms: large range [0x200000000, 0x400000000) (noaccess)\r\n==7839== Warning: set address range perms: large range [0x400000000, 0x600000000) (noaccess)\r\n==7839== Warning: set address range perms: large range [0x400000000, 0x600000000) (noaccess)\r\n==7839== Warning: set address range perms: large range [0x400000000, 0x600000000) (noaccess)\r\n==7839== Warning: set address range perms: large range [0x400000000, 0x600000000) (noaccess)\r\n==7839== Warning: set address range perms: large range [0x400000000, 0x600000000) (noaccess)\r\n==7839== Conditional jump or move depends on uninitialised value(s)\r\n==7839==    at 0xD4190AD: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD44B7CD: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD368ED0: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD4FEA4E: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD34E61B: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD31A25A: cuDevicePrimaryCtxRetain (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xA28FFFF: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA290160: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA29026E: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA291907: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA284EFD: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA277457: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==\r\n==7839== Conditional jump or move depends on uninitialised value(s)\r\n==7839==    at 0xD4190CB: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD44B7CD: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD368ED0: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD4FEA4E: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD34E61B: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD31A25A: cuDevicePrimaryCtxRetain (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xA28FFFF: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA290160: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA29026E: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA291907: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA284EFD: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA277457: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==\r\n==7839== Conditional jump or move depends on uninitialised value(s)\r\n==7839==    at 0xD44B7D9: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD368ED0: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD4FEA4E: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD34E61B: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD31A25A: cuDevicePrimaryCtxRetain (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xA28FFFF: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA290160: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA29026E: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA291907: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA284EFD: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA277457: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA2A4277: cudaMallocPitch (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==\r\n==7839== Use of uninitialised value of size 8\r\n==7839==    at 0xD44B830: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD368ED0: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD4FEA4E: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD34E61B: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD31A25A: cuDevicePrimaryCtxRetain (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xA28FFFF: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA290160: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA29026E: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA291907: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA284EFD: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA277457: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA2A4277: cudaMallocPitch (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==\r\n==7839== Conditional jump or move depends on uninitialised value(s)\r\n==7839==    at 0xD44B84A: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD368ED0: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD4FEA4E: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD34E61B: ??? (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xD31A25A: cuDevicePrimaryCtxRetain (in /usr/lib/x86_64-linux-gnu/libcuda.so.346.96)\r\n==7839==    by 0xA28FFFF: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA290160: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA29026E: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA291907: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA284EFD: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA277457: ??? (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==    by 0xA2A4277: cudaMallocPitch (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcudart.so.7.0.28)\r\n==7839==\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:98) 0\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:98) 1\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:98) 2\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:98) 3\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:98) 4\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:98) 5\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:98) 6\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:98) 7\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:98) 8\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:98) 9\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:98) 10\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:105) Filling for 2nd time\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:118) 11\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:118) 12\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:118) 13\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:118) 14\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:118) 15\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:118) 16\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:118) 17\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:118) 18\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:118) 19\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:118) 20\r\nLOG (UnitTestMatrixRandomizer():nnet-randomizer-test.cc:118) 21\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:144) 0\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:144) 1\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:144) 2\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:144) 3\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:144) 4\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:144) 5\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:144) 6\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:144) 7\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:144) 8\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:144) 9\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:144) 10\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:150) Filling for 2nd time\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:162) 11\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:162) 12\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:162) 13\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:162) 14\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:162) 15\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:162) 16\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:162) 17\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:162) 18\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:162) 19\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:162) 20\r\nLOG (UnitTestVectorRandomizer():nnet-randomizer-test.cc:162) 21\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:193) 0\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:193) 1\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:193) 2\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:193) 3\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:193) 4\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:193) 5\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:193) 6\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:193) 7\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:193) 8\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:193) 9\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:193) 10\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:199) Filling for 2nd time\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:211) 11\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:211) 12\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:211) 13\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:211) 14\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:211) 15\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:211) 16\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:211) 17\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:211) 18\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:211) 19\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:211) 20\r\nLOG (UnitTestStdVectorRandomizer():nnet-randomizer-test.cc:211) 21\r\nTests succeeded.\r\n==7839== Invalid read of size 1\r\n==7839==    at 0x82FA331: cublasDestroy_v2 (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcublas.so.7.0.28)\r\n==7839==    by 0x519BAD: kaldi::CuDevice::~CuDevice() (cu-device.cc:563)\r\n==7839==    by 0xAA26258: __run_exit_handlers (exit.c:82)\r\n==7839==    by 0xAA262A4: exit (exit.c:104)\r\n==7839==    by 0xAA0BECB: (below main) (libc-start.c:321)\r\n==7839==  Address 0x108 is not stack'd, malloc'd or (recently) free'd\r\n==7839==\r\n==7839==\r\n==7839== Process terminating with default action of signal 11 (SIGSEGV)\r\n==7839==  Access not within mapped region at address 0x108\r\n==7839==    at 0x82FA331: cublasDestroy_v2 (in /usr/local/cuda-7.0/targets/x86_64-linux/lib/libcublas.so.7.0.28)\r\n==7839==    by 0x519BAD: kaldi::CuDevice::~CuDevice() (cu-device.cc:563)\r\n==7839==    by 0xAA26258: __run_exit_handlers (exit.c:82)\r\n==7839==    by 0xAA262A4: exit (exit.c:104)\r\n==7839==    by 0xAA0BECB: (below main) (libc-start.c:321)\r\n==7839==  If you believe this happened as a result of a stack\r\n==7839==  overflow in your program's main thread (unlikely but\r\n==7839==  possible), you can try to increase the size of the\r\n==7839==  main thread stack using the --main-stacksize= flag.\r\n==7839==  The main thread stack size used in this run was 8388608.\r\n==7839==\r\n==7839== HEAP SUMMARY:\r\n==7839==     in use at exit: 8,595,792 bytes in 17,730 blocks\r\n==7839==   total heap usage: 21,537 allocs, 3,807 frees, 10,494,220 bytes allocated\r\n==7839==\r\n==7839== LEAK SUMMARY:\r\n==7839==    definitely lost: 48 bytes in 1 blocks\r\n==7839==    indirectly lost: 0 bytes in 0 blocks\r\n==7839==      possibly lost: 1,365,696 bytes in 1,362 blocks\r\n==7839==    still reachable: 7,230,048 bytes in 16,367 blocks\r\n==7839==         suppressed: 0 bytes in 0 blocks\r\n==7839== Rerun with --leak-check=full to see details of leaked memory\r\n==7839==\r\n==7839== For counts of detected and suppressed errors, rerun with: -v\r\n==7839== Use --track-origins=yes to see where uninitialised values come from\r\n==7839== ERROR SUMMARY: 51 errors from 6 contexts (suppressed: 0 from 0)\r\nKilled\r\n```"}}