Anti-Aliasing issue with MSAA, drawing CSG with de

Question

I have reimplemented OpenCSG for modern OpenGL version.

PixelFormatAttributes:

NSOpenGLPFAColorSize    , 24 ,
NSOpenGLPFAAlphaSize    , 8  ,
NSOpenGLPFADepthSize    , 32 ,
NSOpenGLPFAStencilSize  , 8  ,
NSOpenGLPFAAccelerated  ,
NSOpenGLPFADoubleBuffer ,
NSOpenGLPFASupersample  ,
NSOpenGLPFASampleBuffers, 1  ,
NSOpenGLPFASamples      , 4  ,

FBO specs: (tried render to FBO with multisample, but lines getting more strong and visible, look on screenshot at bottom)
- created texture with power of 2, GL_RGBA (tried GL_RGBA8 and GL_RGBA32F)
- GL_DEPTH24_STENCIL8 (tried GL_DEPTH32_STENCIL8, no result)

Simply algorithm Goldfeather:

while (i < depth complexity) {
    take channel for render
       merge layers if no free channel 
    render each layer with stencil func, mask and depth params to channel (FBO)
}
merge layers (taking texture from FBO and render objects again with applying shader below)

In shader I have code for merging (result texture from FBO overlaps on top of rendering for testing, in OpenCSG it's setupProjectiveTexture):

     vec2 ndcPos = gl_FragCoord.xy / sizetexture.xy;
     vec4 maskColor = texture2D(maskTexture, ndcPos.xy);
     if (maskColor[channel] < 0.5) {
        discard;
     }

Looks like after FBO getting not enough clear texture or not right size.

EDIT:
Those lines appears only in overlapping places of subtracting mesh.

EDIT 2:
Fixed with rendering to non MSAA FBO and applying FXAA on result.

Answer 1

Not sure I got your explanations right. What kind of texture are you sampling from in the shader?

I don't know the specifics of the algorithm you're implementing but if that texture contains the results of some previous calculations with multisample gathering already done, I'm really not sure you can use that result for a whole fragment (and all of its samples). The mask is going to fade at edges, which seems to be the problem here.

Maybe you should do sample gathering "by hand", e.g. use a sampler2DMS and read individual samples with texelFetch in the shader.

Another solution might be to set gl_SampleMask to all 1's in the first pass to prevent the fading at borders.