Issue when updating stencil buffer in OpenGL

Question

I'm having issues when drawing using the stencil test in OpenGL on mac. When I first draw the scene, the stencil works fine. I draw a semi-transparent black rectangle in the middle, with writing to the stencil buffer enabled, and then a larger blue rectangle with writing to the stencil buffer disabled. I get the right result when the window first pops up, which looks like this:

However, when I resize the window, and the rendering function gets called again, I get a result which looks like:

or

Sometimes the weird white space follows the middle rectangle, other times the white snaps between seemingly random arrangements, but keeps those arrangements when you go back to that window size. I can find no solution to this online. Here is my render function, which is called any time that the window is resized:

glClearColor(0, 0, 0, 1);
glClearStencil(0x00);
glClear(GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

// Replace data in the stencil buffer with 1s if it passes the test
//  which should be GL_ALWAYS
glStencilFunc(GL_ALWAYS, 1, 0xFF); 
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

// Allow data to be written to the stencil buffer.
glStencilMask(0xFF);

p->fillSquare(1, 1, 1, 0.3, -0.25, 0.25, 0.25, 0.25, -0.25, -0.25, 0.25, -0.25); // Write a semi transparent black rectangle

glStencilMask(0x00); // Disable writing to the stencil buffer.
glStencilFunc(GL_NOTEQUAL, 1, 0xFF); // Only draw if stencil value is 1.

// Draw blue rectangle
p->fillSquare(0.60, 0.60, 0.80, 1, -0.5, 0.5, 0.5, 0.5, -0.5, -0.5, 0.5, -0.5);

glfwSwapBuffers(w);

In case you want to know, p->fillRect() takes four floats for RGBA color of the rectangle and then the x and y coordinate of each vertex.

It seems like there may be some kind of issue in clearing the stencil, but I really can't be sure. I do have the stencil test turned on in my OpenGL initialization function. If you need to know anything else about the other aspects of my code or system, feel free to comment.

Note: I am not using the OpenGL's depth, so it's okay to not be clearing the depth buffer (I've tested this).

Answer 1

The clearing of the stencil buffer does indeed not work (except for the very first time that funtion is called) as you might expect it.

What you missed is that the glStencilMask will also affect glClear(... | GL_STENCIL_BUFFER_BIT). You should move the glStencilMask(0xFF) up a bit, before doing the clear call.