convert window coordinates to 3D world coordinates

Question

I am trying to get the 3D coordinates of a mouse click C++/OpengGL with the glut function glutMouseFunc(). So I created a function like this:

void mouse(int button, int state, int x, int y){
if(button == GLUT_LEFT_BUTTON && state == GLUT_DOWN)
{
    mouse_x=x;
    mouse_y=y;
    }
}

The function gets the window coordinates of the click of the mouse and i use it with the glut function glutMouseFunc() like this:

  glutMouseFunc(mouse);

My question is how would I modify the coordinates given by the mouse function so I could use them in a 3D world. My exact purpose would be the following: to be able to see if I have clicked on a 3D shape drawn in the world.

[EDIT] Would it be easier to transform the coordinates of the 3D object to 2D window coordinates and then compare it to the coordinates of the mouse click?

Answer 1

Mouse click does not correspond to a point in 3d space, but to a ray.

In any case, you use gluUnProject.

If you know scene "depth" under cursor, then you can get 3d position of a click - by passing depth via winZ parameter.

If you don't know depth, pass 0.0 in winZ parameter, to get start of the ray, and 1.0 to get the "end". You'll have to calculate yourself if this ray hits anything.