cudaErrorCudartUnloading (error 29) due to "driver shutting down"
<p>I have recently installed opencv from the <a href="https://github.com/opencv/opencv" rel="nofollow noreferrer">github repo</a> and compiled with CUDA support. I' planning on updating a company framework from CPU to GPU version, and I've run into a recurring issue:</p>

<pre><code>========= Program hit cudaErrorCudartUnloading (error 29) due to "driver shutting down" on CUDA API call to cudaStreamSynchronize. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2eeda3]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcudart.so.7.5 (cudaStreamSynchronize + 0x15e) [0x421ce]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0x29a909]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 (_ZN2cv4cuda6Stream4ImplD1Ev + 0x6e) [0x29c0ee]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0x29c12a]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0x29a886]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0x29d08e]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__cxa_finalize + 0x8f) [0x37eaf]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0xbb033]
=========
========= Program hit cudaErrorCudartUnloading (error 29) due to "driver shutting down" on CUDA API call to cudaSetDevice. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2eeda3]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcudart.so.7.5 (cudaSetDevice + 0x180) [0x2fa30]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0x29d0fd]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__cxa_finalize + 0x8f) [0x37eaf]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0xbb033]
=========
========= Program hit cudaErrorCudartUnloading (error 29) due to "driver shutting down" on CUDA API call to cudaFree. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2eeda3]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcudart.so.7.5 (cudaFree + 0x186) [0x3c666]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0x29d0cd]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__cxa_finalize + 0x8f) [0x37eaf]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0xbb033]
=========
========= Program hit cudaErrorCudartUnloading (error 29) due to "driver shutting down" on CUDA API call to cudaSetDevice. 
=========     Saved host backtrace up to driver entry point at error
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcuda.so.1 [0x2eeda3]
=========     Host Frame:/usr/lib/x86_64-linux-gnu/libcudart.so.7.5 (cudaSetDevice + 0x180) [0x2fa30]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0x29d0fd]
=========     Host Frame:/lib/x86_64-linux-gnu/libc.so.6 (__cxa_finalize + 0x8f) [0x37eaf]
=========     Host Frame:/usr/local/lib/libopencv_core.so.3.1 [0xbb033]
=========
========= LEAK SUMMARY: 0 bytes leaked in 0 allocations
========= ERROR SUMMARY: 4 errors
</code></pre>

<p>What is this, or how could I trace it?
When running the application from the terminal I get no errors, exceptions or segmentation faults.</p>

<h2>EDIT:</h2>

<pre><code>#include &lt;opencv2/imgproc/imgproc.hpp&gt;
#include &lt;opencv2/cudaimgproc.hpp&gt;
#include &lt;opencv2/core/cuda.hpp&gt;
#include &lt;opencv2/cudaobjdetect.hpp&gt;
#include &lt;opencv2/cudafeatures2d.hpp&gt;    
#include &lt;iostream&gt;
#include &lt;stdio.h&gt;
#include &lt;chrono&gt;

int detectAndDisplay(
                     cv::Mat frame, 
                     cv::Ptr&lt;cv::cuda::CascadeClassifier&gt; &amp; cascade_gpu
                    )
{
    cv::cuda::GpuMat image_src, image_grey;
    std::vector&lt;cv::Rect&gt; faces;
    image_src.upload(frame);
    cv::cuda::cvtColor(image_src, image_grey, CV_BGR2GRAY);
    cv::cuda::equalizeHist(image_grey, image_grey);
    cascade_gpu-&gt;setFindLargestObject(true);
    cascade_gpu-&gt;setScaleFactor(1.23);
    cascade_gpu-&gt;setMinNeighbors(3);
    cascade_gpu-&gt;setMinObjectSize(cv::Size(0, 0));
    cascade_gpu-&gt;setMaxObjectSize(image_src.size());
    cv::cuda::GpuMat buffer;
    cascade_gpu-&gt;detectMultiScale(image_grey, buffer);
    cascade_gpu-&gt;convert(buffer, faces);
    return faces.size();
}

int main( int argc, const char* argv[] )
{
    cv::String face_cascade_name = "/usr/share/opencv/haarcascades/haarcascade_frontalface_alt.xml";
    cv::Mat image;
    image = cv::imread("catniss.jpeg", CV_LOAD_IMAGE_COLOR);
    cv::Ptr&lt;cv::cuda::CascadeClassifier&gt; cascade_gpu;
    cascade_gpu = cv::cuda::CascadeClassifier::create(face_cascade_name);
    for (int j = 0; j &lt; 100; j++) {
        detectAndDisplay(image, cascade_gpu); 
    }
    return 0;
}
</code></pre>

<p>A minimal <code>CMakeLists.txt</code> for your convenience:</p>

<pre><code>cmake_minimum_required(VERSION 2.6)
project(face_detection)
add_executable(face_detection source/main.cpp)
find_package(OpenCV REQUIRED)
target_link_libraries(face_detection  ${OpenCV_LIBS})
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}-std=c++1y")
</code></pre>

<p>Follow the typical build phase, and then just run</p>

<pre><code>cuda-memcheck --leak-check full ./face_detection
</code></pre>

<p>I have a Debian <code>jessie-backports</code> install of CUDA 7.5 with respective drivers, running on two GTX 660.</p>