Unity3D and OpenCV integration and passing each other camera frames
<p><strong>Background:</strong></p>

<p>I've been working on an AR project using Unity3D and OpenCV. The integration has been done in the following way.</p>

<p>Our OpenCV program and the OpenCV library itself compiles into a single dynamic library (<code>so</code> file Linux). (OpenCV library is linked to statically)</p>

<p>Then, in Unity3D, in a C# script we import it (<code>DllImport</code>) and use its functionalities.</p>

<p>So far everything looks clear, but the problem is here: Both parts (Unity3D and OpenCV) need every frames of the camera, so there are multiple ways to give them camera frames:</p>

<ol>
<li><p>Unity3D reads frames (<code>WebCamTexture</code>) and sends them via converting to three 2D arrays and passing to OpenCV part. (In this way, OpenCV part does not have access to camera)</p></li>
<li><p>OpenCV part reads frames (<code>VideoCapture::open</code>) and sends back to Unity3D. (In this way, Unity3D does not have access to camera)</p></li>
<li><p>Both parts are allowed to open the camera.</p></li>
</ol>

<p>Note that the major processing is done in the OpenCV part and Unity3D only needs frames to show a camera preview.</p>

<p><strong>Question:</strong></p>

<p>It's all about performance! Which approach is the best way in performance and being the best-practice point of view?</p>