LibGDX Stencil Buffers while using SpriteBatch

Question

This is a continuation of my previous problem and post, seen here. Thanks to the answer I received there I feel I was able to get a little closer to my goal, as well as further my learning of OpenGL, but shortly after figuring out the basics of working with stencil buffers, I've run into a problem.

It seems that when I draw a sprite to the stencil buffer, it draws the entire square area, rather than just the pixels that aren't fully transparent as I had ignorantly hoped. I vaguely understand why it happens that way, but I am not sure where the solution lies. I have experimented with the stencil itself quite a bit, and I have modified the shaders that the spritebatch uses to discard low-alpha fragments, but I seem to be failing to see the bigger picture.

As a visual example of the problem, I will continue with the examples I used in the previous question. Right now, trying to draw two circles over each other (So they blend perfectly, no overlapping), I am getting this :

So, basically, is there a way for me to utilize stencil buffers using the Sprite and SpriteBatch functionality of LibGDX on complex shapes (Circles are only being used as an example), or do I need to look for an alternative route?

EDIT ::

    Gdx.gl.glClearColor(1, 1, 1, 1);
    Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT | GL20.GL_DEPTH_BUFFER_BIT);

    Gdx.gl.glEnable(GL20.GL_BLEND);
    Gdx.gl.glBlendFunc(GL20.GL_SRC_ALPHA, GL20.GL_ONE_MINUS_SRC_ALPHA);
    Gdx.gl.glEnable(GL20.GL_DEPTH_TEST);
    Gdx.gl.glDepthMask(true);

    batch.begin();
    sprite.draw(batch);
    sprite2.draw(batch);
    batch.end();

    Gdx.gl.glDisable(GL20.GL_DEPTH_TEST);

Answer 1

Mby this helps. I solve same problem with blending and depth test. First you need clear depth and color bit.

Gdx.gl.glClear(GL10.GL_COLOR_BUFFER_BIT | GL10.GL_DEPTH_BUFFER_BIT);

Then you need enable some gl features:

Gdx.gl.glEnable(GL20.GL_BLEND);
Gdx.gl.glBlendFunc(GL20.GL_SRC_ALPHA, GL20.GL_ONE_MINUS_SRC_ALPHA);
Gdx.gl.glEnable(GL10.GL_DEPTH_TEST);
Gdx.gl.glDepthMask(true);

After that you can draw your objects. In my project smtng like this:

shapeRenderer.setColor(c.getR(), c.getG(), c.getB(), FIELD_ALPHA);
shapeRenderer.filledCircle(p.getPos().x + s.getOffset().x, ApplicationEnv.SCREEN_HEIGHT - p.getPos().y + s.getOffset().x, b.getRadius());

Important! After drawing you call end() for your spriteBatch (or something else) and disable depth_test feature:

Gdx.gl.glDisable(GL10.GL_DEPTH_TEST);

Result: Before After

Answer 2

Stencil writing/testing happens at the fragment level. When you draw a circle, you are actually drawing a quad of fragments where each fragment may or may not be textured. The issue of the matter is that the GPU doesn't care what color you write into the fragment when it does the stencil test for the fragment. Therefore, you need to discard fragments to stop the stencil writes for the parts of the quad where you don't want to write stencil values.

TL;DR Use "if(gl_FragColor.a < 0.01) discard;" in the fragment shader to make sure a circle of fragments (and stencil values) are generated.