opengl - how to implement "portal rendering"
<p>I've been trying to implement something like the game <a href="https://en.wikipedia.org/wiki/Antichamber" rel="noreferrer">Antichamber</a> (to be more precisly this trick shown below) for the past week: </p>

<p><a href="https://i.stack.imgur.com/XRsPh.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/XRsPh.jpg" alt="trick"></a></p>

<p>Here's a video of what I'm hoping to achieve (even though it was done with the Unreal Engine 4; I'm not using that): <a href="https://www.youtube.com/watch?v=Of3JcoWrMZs" rel="noreferrer">https://www.youtube.com/watch?v=Of3JcoWrMZs</a></p>

<p>I looked up the best way to do this and I found out about the stencil buffer. Between <a href="https://th0mas.nl/2013/05/19/rendering-recursive-portals-with-opengl/" rel="noreferrer">this</a> article and <a href="https://github.com/SimonLarsen/glPortal/blob/master/Main.cpp" rel="noreferrer">this</a> code (the "drawPortals()" function) I found online I managed to almost implement it.</p>

<p>It works nicely with one portal to another room (not a crossable portal, meaning you can't walk through it and be teleported in the other room). In my example I'm drawing a portal to a simple squared room with a sphere in it; behind the portal there is another sphere, that I used to check whether the depth buffer was working correctly and drawing it behind the portal:</p>

<p><a href="https://i.stack.imgur.com/UTLs1.png" rel="noreferrer"><img src="https://i.stack.imgur.com/UTLs1.png" alt="front"></a></p>

<p><a href="https://i.stack.imgur.com/EKn9M.png" rel="noreferrer"><img src="https://i.stack.imgur.com/EKn9M.png" alt="side"></a></p>

<p>The problems arise when I add another portal that is near this one. In that case I manage to display the other portal correctly (the lighting is off, but the sphere on the right is of a different color to show that it's another sphere):</p>

<p><a href="https://i.stack.imgur.com/poJWz.png" rel="noreferrer"><img src="https://i.stack.imgur.com/poJWz.png" alt="enter image description here"></a></p>

<p>But if I turn the camera so that the first portal has to be drawn over the second one, then the depth of the first one becomes wrong, and the second portal gets drawn over the first one, like this:</p>

<p><a href="https://i.stack.imgur.com/YuW0S.png" rel="noreferrer"><img src="https://i.stack.imgur.com/YuW0S.png" alt="enter image description here"></a></p>

<p>while it should be something like this:</p>

<p><a href="https://i.stack.imgur.com/IGpFu.png" rel="noreferrer"><img src="https://i.stack.imgur.com/IGpFu.png" alt="enter image description here"></a></p>

<p>So, this is the problem. I'm probably doing something wrong with the depth buffer, but I can't find what.</p>

<p>My code for the rendering part is pretty much this:</p>

<pre><code>glClear(GL_DEPTH_BUFFER_BIT);
glEnable(GL_STENCIL_TEST);

// First portal
glPushMatrix();

// Disable writing to the color and depht buffer; disable depth testing
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);
glDisable(GL_DEPTH_TEST);

// Make sure that the stencil always fails
glStencilFunc(GL_NEVER, 1, 0xFF);

// On fail, put 1 on the buffer
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);

// Enable writing to the stencil buffer
glStencilMask(0xFF);

// Clean the buffer
glClear(GL_STENCIL_BUFFER_BIT);

// Finally draw the portal's frame, so that it will have only 1s in the stencil buffer; the frame is basically the square you can see in the pictures
portalFrameObj1.Draw();

/* Now I compute the position of the camera so that it will be positioned at the portal's room; the computation is correct, so I'm skipping it */

// I'm going to render the portal's room from the new perspective, so I'm going to need the depth and color buffers again
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);
glEnable(GL_DEPTH_TEST);
glClear(GL_DEPTH_BUFFER_BIT);

// Disable writing to the stencil buffer and enable drawing only where the stencil values are 1s (so only on the portal frame previously rendered)
glStencilMask(0x00);
glStencilFunc(GL_EQUAL, 1, 0xFF);

// Draw the room from this perspective
portalRoomObj1.Draw();

glPopMatrix();


// Now the second portal; the procedure is the same, so I'm skipping the comments
glPushMatrix();
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_TRUE);
glDepthMask(GL_FALSE);
glDisable(GL_DEPTH_TEST);

glStencilFunc(GL_NEVER, 1, 0xFF);
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);
glStencilMask(0xFF);
glClear(GL_STENCIL_BUFFER_BIT);

portalFrameObj2.Draw();

/* New camera perspective computation */

glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);
glEnable(GL_DEPTH_TEST);
glClear(GL_DEPTH_BUFFER_BIT);

glStencilMask(0x00);
glStencilFunc(GL_EQUAL, 1, 0xFF);

portalRoomObj2.Draw();

glPopMatrix();


// Finally, I have to draw the portals' frames once again but this time on the depth buffer, so that they won't get drawn over; first off, disable the stencil buffer
glDisable(GL_STENCIL_TEST);

// Disable the color buffer
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);

glClear(GL_DEPTH_BUFFER_BIT);

// Draw portals' frames
portalFrameObj1.Draw();
portalFrameObj2.Draw();

// Enable the color buffer again
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);


/* Here I draw the rest of the scene */
</code></pre>

<p><strong>UPDATE</strong> </p>

<p>I managed to find out what the problem is, but I'm still not able to solve it. It isn't related to the depth buffer actually, but to the stencil.</p>

<p>Basically, the way I'm drawing the first portal is this:
1) Fill the portal frame's bits in the stencil buffer with 1s; outside the portal there are only 0s
2) Draw the portal room where the stencil has 1s (so that it gets drawn onto the frame's portal</p>

<p>And I repeat this for the second portal.</p>

<p>For the first portal, I get at step 1 something like this then (pardon the stupid drawings, I'm lazy):
<a href="https://i.stack.imgur.com/Hdc8q.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Hdc8q.png" alt="enter image description here"></a></p>

<p>Then after step 2:<br>
<a href="https://i.stack.imgur.com/lhl5K.png" rel="noreferrer"><img src="https://i.stack.imgur.com/lhl5K.png" alt="enter image description here"></a></p>

<p>Then I start with the second portal:<br>
<a href="https://i.stack.imgur.com/PTcrX.png" rel="noreferrer"><img src="https://i.stack.imgur.com/PTcrX.png" alt="enter image description here"></a></p>

<p>But now, between step 1 and 2, I tell the stencil to draw only where the bits are 1s; since the buffer is now cleared, I lost track of the 1s of the first portal, so if part of the second portal's frame is behind the previous frame, the 1s of the second frames will still be drawn with the second portal's room, regardless of the depth buffer. Kind of like in this image:
<a href="https://i.stack.imgur.com/9oNpF.png" rel="noreferrer"><img src="https://i.stack.imgur.com/9oNpF.png" alt="enter image description here"></a></p>

<p>I don't know if I managed to explain it right...</p>