How does mask affect stencil value according to st

The documentation in OpenGL reference pdf (both OpenGL 3.3 and 4.5 specs) is not much clear about what happens to the stored stencil value when a mask is applied.

In example If I have the following mask:

glStencilMask( 0x06);

and stored in the stencil buffer there is already this value:

0x06

If the stencil operation is GL_INCR_WRAP

what should happens when StencilOp is correctly invoked on that pixel?

Basically I have the mask:

00000110

and the value

00000110

and I try to increment it, is it wrapped?

00000010

or is it just zeroed? (00000110 + 1) & mask

00000000

Section 17.4.2 "Fine Control of Buffer Updates" of the OpenGL 4.5 core profile specification states:

The commands void StencilMask( uint mask ); void StencilMaskSeparate( enum face, uint mask ); control the writing of particular bits into the stencil planes. The least significant s bits of mask, where s is the number of bits in the stencil buffer, specify an integer mask. Where a 1 appears in this mask, the corresponding bit in the stencil buffer is written; where a 0 appears, the bit is not written.

The glStencilMask() parameter controls which bit planes are written to the stencil buffer. It does not control what is read, or how glStencilOp operates.

Section 17.3.5 "Stencil Test" states (my emphasis):

For purposes of increment and decrement, the stencil bits are considered as an unsigned integer. Incrementing or decrementing with saturation clamps the stencil value at 0 and the maximum representable value. Incrementing or decrementing without saturation will wrap such that incrementing the maximum representable value results in 0, and decrementing 0 results in the maximum representable value.

The stencil mask itself is not relevant in that stage of the pipeline. It is only applied when the fragment is finally written to the framebuffer, like all gl*Mask() functions.

So having the value 0110 in the buffer and applying GL_INCR_WRAP leads to 0111 and when this is written the buffer, the mask is applied, so you basically end up with 0110 again (and not 0).

Also note that there is also a mask parameter in glStencilFunc() defining a bit mask to be applied before the stencil test. Quoting section 17.3.5 again: (my emphasis):

StencilFunc and StencilFuncSeparate take three arguments that control whether the stencil test passes or fails. ref is an integer reference value that is used in the unsigned stencil comparison. Stencil comparison operations and queries of ref clamp its value to the range [0; 2^s - 1], where s is the number of bits in the stencil buffer attached to the draw framebuffer. The s least significant bits of mask are bitwise ANDed with both the reference and the stored stencil value, and the resulting masked values are those that participate in the comparison controlled by func.

So if you want to wrap around a at some value 2^n-1, you can simply ignore the additional bits in the stencil buffer and just test these bits in the stencil test.