How do I reverse-project 2D points into 3D?

Question

I have 4 2D points in screen-space, and I need to reverse-project them back into 3D space. I know that each of the 4 points is a corner of a 3D-rotated rigid rectangle, and I know the size of the rectangle. How can I get 3D coordinates from this?

I am not using any particular API, and I do not have an existing projection matrix. I'm just looking for basic math to do this. Of course there isn't enough data to convert a single 2D point to 3D with no other reference, but I imagine that if you have 4 points, you know that they're all at right-angles to each other on the same plane, and you know the distance between them, you should be able to figure it out from there. Unfortunately I can't quite work out how though.

This might fall under the umbrella of photogrammetry, but google searches for that haven't led me to any helpful information.

Answer 1

D. DeMenthon devised an algorithm to compute the pose of an object (its position and orientation in space) from feature points in a 2D image when knowing the model of the object -- this is your exact problem:

We describe a method for finding the pose of an object from a single image. We assume that we can detect and match in the image four or more noncoplanar feature points of the object, and that we know their relative geometry on the object.

The algorithm is known as Posit and is described in it classical article "Model-Based Object Pose in 25 Lines of Code" (available on its website, section 4).

Direct link to the article: http://www.cfar.umd.edu/~daniel/daniel_papersfordownload/Pose25Lines.pdf OpenCV implementation: http://opencv.willowgarage.com/wiki/Posit

The idea is to repeatedly approximating the perspective projection by a scaled orthographic projection until converging to an accurate pose.

Answer 2

Assuming that the points are indeed part of a rectangle, I'm giving a generic idea :

Find two points with max inter-distance: these most probably define a diagonal (exception: special cases where the rectangle is almost paralell to the YZ plane, left for the student). Call them A, C. Calculate the BAD, BCD angles. These, compared to right angles, give you orientation in 3d space. To find out about z distance, you need to correlate the projected sides to the known sides, and then, based on the 3d projection method (is it 1/z?) you're on the right track to know distances.