next up previous contents
Next: 8.7 Compositing Images with Up: 8. Blending and Compositing Previous: 8.5 Blending Transitions

8.6 The Stencil Buffer

The stencil buffer is like the depth and color buffers, except stencil pixels don't represent colors or depths, but have application-specific meanings. The stencil buffer isn't directly visible like the color buffer, but the bits in the stencil planes form an unsigned integer that affects and is updated by drawing commands, through the stencil function and the stencil operations. The stencil function controls whether a fragment is discarded or not by the stencil test, and the stencil operation determines how the stencil planes are updated as a result of that test [67].

Stencil buffer actions are part of OpenGL's fragment operations. Stencil testing occurs immediately after the alpha test, and immediately before the depth test. If GL_STENCIL_TEST is enabled, and stencil planes are available, the application can control what happens under three different scenarios:

tex2html_nowrap 3.
The stencil test fails.
tex2html_nowrap 4.
The stencil test passes, but the depth test fails.
tex2html_nowrap 5.
Both the stencil and the depth test pass.

Whether a stencil operation for a given fragment passes or fails has nothing to do with the color or depth value of the fragment. The stencil operation is a comparison between the value in the stencil buffer for the fragment's destination pixel and the stencil reference value. A mask is bitwise AND-ed with the value in the stencil planes and with the reference value before the comparison is applied. The reference value, the comparison function, and the comparison mask are set by glStencilFunc(). The comparison functions available are listed in Table 3.

Stencil function and stencil test are often used interchangeably in these notes, but the ``stencil test'' specifically means the application of the stencil function in conjunction with the stencil mask.

Table 3: Stencil Buffer Comparisons
Comparison Description of comparison test between reference and stencil value
GL_NEVER always fails
GL_ALWAYS always passes
GL_LESS passes if reference value is less than stencil buffer
GL_LEQUAL passes if reference value is less than or equal to stencil buffer
GL_EQUAL passes if reference value is equal to stencil buffer
GL_GEQUAL passes if reference value is greater than or equal to stencil buffer
GL_GREATER passes if reference value is greater than stencil buffer
GL_NOTEQUAL passes if reference value is not equal to stencil buffer

If the stencil test fails, the fragment is discarded (the color and depth values for that pixel remain unchanged) and the stencil operation associated with the stencil test failing is applied to that stencil value. If the stencil test passes, then the depth test is applied. If the depth test passes (or if depth testing is disabled or if the visual does not have a depth buffer), the fragment continues on through the pixel pipeline, and the stencil operation corresponding to both stencil and depth passing is applied to the stencil value for that pixel. If the depth test fails, the stencil operation set for stencil passing but depth failing is applied to the pixel's stencil value.

Thus, the stencil test controls which fragments continue towards the framebuffer, and the stencil operation controls how the stencil buffer is updated by the results of both the stencil test and the depth test.

The stencil operations available are described in Table 4. The GL_INCR and GL_DECR operations saturate so incrementing the maximum stencil value is still the maximum value and decrementing the value zero is still zero. Some implementations support the EXT_stencil_wrap extension which adds two new stencil operations, GL_INCR_WRAP_EXT and GL_DECR_WRAP_EXT. These operations ``wrap'' such that incrementing the maximum stencil value generates zero and decrementing zero generates the maximum value.

Table 4: Stencil Buffer Operations
Stencil Operation Results of Operation on Stencil Values
GL_KEEP stencil value unchanged
GL_ZERO stencil value set to zero
GL_REPLACE stencil value replaced by stencil reference value
GL_INCR stencil value incremented
GL_DECR stencil value decremented
GL_INVERT stencil value bitwise inverted

The glStencilOp() call sets the stencil operations for all three stencil test results: stencil fail, stencil pass/depth buffer fail, and stencil pass/depth buffer pass.

Writes to the stencil buffer can be disabled and enabled per bit by glStencilMask(). This allows an application to apply stencil tests without the results affecting the stencil values. Keep in mind, however, that the GL_INCR and GL_DECR operations operate on each stencil value as a whole, and may not operate as expected when the stencil mask is not all ones. Stencil writes can also be disabled by calling glStencilOpGL_KEEP, GL_KEEP, GL_KEEP(GL_KEEP, GL_KEEP, GL_KEEP).

There are three other important ways of controlling and accessing the stencil buffer. Every stencil value in the buffer can be set to a desired value by calling glClearStencil() and glClearGL_STENCIL_BUFFER_BIT(GL_STENCIL_BUFFER_BIT). The contents of the stencil buffer can be read into system memory using glReadPixels() with the format parameter set to GL_STENCIL_INDEX. The contents of the stencil buffer can also be set using glDrawPixels().

Different machines support different numbers of stencil bits per pixel. Table 5 lists the number of stencil bits supported by a few selected OpenGL implementations. Use glGetIntegervGL_STENCIL_BITS, ...(GL_STENCIL_BITS, ...) to see how many bits are available. If multiple stencil bits are available, glStencilMask() and the mask argument to glStencilFunc() can be used to divide up the stencil buffer into a number of different sections. This allows the application to store separate stencil values per pixel within the same stencil buffer.

Table 5: Stencil Bits Supported by Selected OpenGL Implementations
OpenGL Implementation Stencil Bits Supported
Most software implementations 8
(Mesa, Microsoft OpenGL, SGI OpenGL for Windows)  
3Dlabs Permedia II 1
SGI Indigo2 Extreme 4
SGI Octane MXI 8
ATI Rage 128 8 (32-bit mode only)
NVIDIA RIVA TNT 8 (32-bit mode only)
SGI Onyx2 InfiniteReality 1 or 8 (multisampled!)

The following sections describe how to use the stencil buffer in a number of useful multipass rendering techniques.

next up previous contents
David Blythe