Calculate surface normals from depth image using neighboring pixels cross product
<p>As the title says I want to calculate the surface normals of a given depth image by using the cross product of neighboring pixels. I would like to use Opencv for that and avoid using PCL however, I do not really understand the procedure, since my knowledge is quite limited in the subject. Therefore, I would be grateful is someone could provide some hints. To mention here that I do not have any other information except the depth image and the corresponding rgb image, so no <code>K</code> camera matrix information.</p>

<p>Thus, lets say that we have the following depth image:</p>

<p><a href="https://i.stack.imgur.com/Air7G.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Air7G.png" alt="enter image description here"></a></p>

<p>and I want to find the normal vector at a corresponding point with a corresponding depth value like in the following image:</p>

<p><a href="https://i.stack.imgur.com/ISL4l.png" rel="noreferrer"><img src="https://i.stack.imgur.com/ISL4l.png" alt="enter image description here"></a> </p>

<p>How can I do that using the cross product of the neighbouring pixels? I do not mind if the normals are not highly accurate.</p>

<p>Thanks.</p>

<hr>

<p><strong>Update:</strong></p>

<p>Ok, I was trying to follow @timday's answer and port his code to Opencv. With the following code:</p>

<pre><code>Mat depth = &lt;my_depth_image&gt; of type CV_32FC1
Mat normals(depth.size(), CV_32FC3);

for(int x = 0; x &lt; depth.rows; ++x)
{
    for(int y = 0; y &lt; depth.cols; ++y)
    {

        float dzdx = (depth.at&lt;float&gt;(x+1, y) - depth.at&lt;float&gt;(x-1, y)) / 2.0;
        float dzdy = (depth.at&lt;float&gt;(x, y+1) - depth.at&lt;float&gt;(x, y-1)) / 2.0;

        Vec3f d(-dzdx, -dzdy, 1.0f);
        Vec3f n = normalize(d);

        normals.at&lt;Vec3f&gt;(x, y) = n;
    }
}

imshow("depth", depth / 255);
imshow("normals", normals);
</code></pre>

<p>I am getting the correct following result (I had to replace <code>double</code> with <code>float</code> and <code>Vecd</code> to <code>Vecf</code>, I do not know why that would make any difference though):</p>

<p><a href="https://i.stack.imgur.com/SYhcY.png" rel="noreferrer"><img src="https://i.stack.imgur.com/SYhcY.png" alt="enter image description here"></a></p>