Back face culling for linestrips

Question

I have circle in 3D space (red on image) with normals (white)

This circle is being drawn as linestrip.

Problem is: i need to draw only those pixels whose normals directed into camera (angle between normal and camera vector is < 90) using discard in fragment shader code. Like backface culling but for lines.

Red part of circle is what i need to draw and black is what i need to discard in fragment shader.

Good example is 3DS Max rotation gizmo, back sides of lines are hidden:

So, in fragment shader i have:

if(condition)
    discard;

Help me to come up with this condition. Considering both orthographic and perspective cameras would be good.

Answer 1

Well, you already described your condition:

(angle between normal and camera vector is < 90)

You have to forward your normals to the fragment shader (don't forget to re-normalize it in the FS, the interpolation will change the length). And you need the viewing vector (in the same space than your normals, so you might transform normal to eye space, or use world space, or even transform the view direction/camera location into object space). Since the condition angle(N,V) >= 90 (degrees) is the same as cos(angle(N,V)) <= 0 (assuming normalized vectors), you can simply use the dot product:

if (dot(N,V) <= 0)
    discard;

UPDATE:

As you pointed out in the comments, you have the "classical" GL matrices available. So it makes sense to do this transformation in eye space. In the vertex shader, you put

in vec4 vertex;  // object space position
in vec3 normal;  // object space normal direction
out vec3 normal_eyespace;
out vec3 vertex_eyespace;

uniform mat3 normalMatrix;
uniform mat4 modelView;
uniform mat4 projection;

void main()
{
    normal_eyespace = normalize(normalMatrix * normal);
    vec4 v = modelViewMatrix * vertex;
    vertex_eyespace = v.xyz;
    gl_Position=projectionMatrix * v;
}

and in the fragment shader, you can simply do

in vec3 normal_eyespace;
in vec3 vertex_eyespace;

void main()
{
    if (dot(normalize(normal_eyespace), normalize(-vertex_eyespace)) <= 0)
        discard;
    // ...
}

Note: this code assumes modern GLSL with in/out instead of attribute/varying qualifiers. I also assume no builtin attributes. But that code should be easily adaptable to older GL.