Calculating camera ray direction to 3d world pixel

I'm want to calculate ray directions from camera to pixel positions (in world coords) with given height as described in this paper. Camera image size is 640,480. I calibrated the intrinsic camera parameters and undistort each image. I measured the physical distance between camera and background plane (29 cm) as sketched here:

1 cm translates to 25 pixel (assuming quadratic pixels). My first approach was to calculate based on pixel- and camera position as follows:

float3 pixPos = (float3)(u, v, z);
float3 camPos = (float3)(height/2, width/2, 29*25);
float3 ray = normalize(pixPos-camPos);

where u,v are image coords from 0,0 to height,width and z are my (already estimated) height values. This doesn't seem to be the right way. I already did a search on SO and found this answer, but the solution described there doesn't include pixel heights (z here).

I was solving same problem few years back, when I was writing my diploma thesis. Here is a part of it which describes how to create camera for raytracing..

First of all you need to define coordinate system of camera. It's orthonormal coordinate system, that means that all three basis vectors are perpendicular to each other and have same size (doesn't have to be "one"). Also you need to point out, if your coordinate system of camera is right-handed or left-handed (I will talk about left-handed). First you'll need to define up vector (the vector shows you what is y axis on your screen) and camera direction vector (so vector from eye position to middle of your projection plane). Then you'll calculate left vector (or right vector if you need) (which points where exactly is x axis on your screen) by cross product of the up vector and camera direction vector. Now, because only camera direction vector and left vector are perpendicular for sure, you have to make one more cross product of camera direction vector and left vector (vector y on the image).

So you'll get camera coordination system like this one

No you have to define, how big is your projection screen in world coordinates. That might be sometimes tricky, so you can also define it by two angles (phi and theta) and distance from eye position (lets call it d).

You'll get and . (the x vector is the vector which defines x axis on screen, so it's left vector or right vector, depends of handedness) By linear combination of those two vectors u and v you can compute any position on your projection screen. Coefficients alpha and though represents distances the point from middle of projection screen.