I am attempting to recreate a classical shape from shading algorithm seen in the Trucco/Verri text "Introductory Techniques for 3d Computer Vision", but I am having a hard time understanding the fft function in matlab. Essentially, I need to use the integrability constraint to get the depth (Z) of an image. I am not sure when to use fftshift or not in this scenario. Here is the code I have so far. Based on http://www.mathworks.com/matlabcentral/newsreader/view_thread/285244 I basically wrapped all my fft2s in fftshifts, but I don't think this is the correct usage. Could someone please explain to me the usage and what I am doing wrong? Thank you. Basically, I'm trying to take my p and q (values which are the updates based on pixel intensities) transform them into the Fourier domain so as to use them in the equation C. Then I want to transform the Equation C back to the time domain because that will give me Z the depth. I also want to update P and Q based on C in the Fourier Domain.

    wx = (2.* pi .* x) ./ m; 
    wy = (2.* pi .* y) ./ n; 
    wx = ifftshift(wx); wy=ifftshift(wy);

    Cp = fftshift(fft2(fftshift(p))); 
    Cq = fftshift(fft2(fftshift(q)));
    C = -1i.*(wx .* Cp + wy .* Cq)./(wx.^2 + wy.^2); 
    Z = abs((ifft2(ifftshift(C)))); 
    p = ifftshift(ifft2(ifftshift(1i * wx .* C))); 
    q = ifftshift(ifft2(ifftshift(1i * wy .* C)));

This is a tricky question because in general there is not a right answer. There may be some wrong answers though. I will try to explain. If the answer get a little bit too wordy, you can always just skip down to the summary section and see if it helps.

Gotchas

Gotcha #1:

When you use Matlab's fft (or in your case fft2) function, the first element of the output (in your case X(1,1)) represents the DC bias. If you subsequently call fftshift on your output, everything gets shifted around in a way that places the DC bias at the center. In the 2-dimensional case, it looks something like this:

Notice that the point that was at the top-left corner of block 1 gets moved to the center. While this is a perfectly valid representation of the data, we have to be careful because we have changed the meaning of the (1,1) bin. If I were to attempt an inverse transform at this point, the output would be wrong!

B = ifft2(fft2(A));            % B is equal to A
C = ifft2(fftshift(fft2(A)));  % C is not equal to A

Gotcha #2:

The ifftshift function should be thought of as the inverse of the fftshift operation. It should not be thought of as a shift that applies to ifft operation. For this reason, I feel that the function names are very misleading.

In my experience, it is most common for an ifftshift to precede an fft/ifft function, and for an fftshift to follow fft/ifft function. In fact, I would go so far as to say that if you ever find yourself doing one of the following things, you have probably made a mistake:

B = ifftshift(ifft(A));        % Don't do this
C = fft(fftshift(A));          % Don't do this either

The following helpful note is found in the Matlab documentation for ifftshift

Note: ifftshift will undo the results of fftshift. If the matrix X contains an odd number of elements, ifftshift(fftshift(X)) must be done to obtain the original X. Simply performing fftshift(X) twice will not produce X.

For example:

B = ifftshift(fftshift(A));    % B is equal to A
C = fftshift(fftshift(A));     % C is not equal to A

Gotcha #3:

The DFT has many interesting properties, one of which is that the DFT of a real, even sequence is real and even. We can often use this fact as a simple sanity check. If we put a real, even sequence into the fft function and get back something back that is not real and even, we have a problem.

We must take careful note of what an even function looks like when it comes to the DFT. The sequence 3 2 1 0 1 2 3 appears to be even, right? The left half is a mirror image of the right half. This would be true if the fourth element of the sequence represented t=0. However, because of the way the FFT algorithm is set up, the first element always represents the t=0 element.

We can remedy the problem by performing an ifftshift operation before the FFT in order to shift the center to the first element. Note that for a sequence with even length, the element x[N/2+1] is assumed to be the center.

A1 = [ 3 2 1 0 1 2 3 ];        % A1 real, even sequence about A1(4)
B1 = fft(ifftshift(A1));       % B1 is a real, even sequence
C1 = fft(A1);                  % C1 is _not_ a real, even sequence
abs(B1) == abs(C1)             % B1 and C1 differ only in phase

A2 = [ 0 1 2 3 3 2 1 ];        % A2 real, even sequence about A2(0)
B2= fft(ifftshift(A2));        % B2 is _not_ a real, even sequence
C2= fft(A2);                   % C2 is a real, even sequence
abs(B2) == abs(C2)             % B2 and C2 differ only in phase

As you can see by the last example, it would be incorrect to say "always use ifftshift before fft." What if the first element of my data is already the t=0 element? Then applying ifftshift would be the wrong thing to do.

Summary

In general, ifftshift should only be used before applying an fft/ifft. The fft and ifft functions always assume that the first element of your data represents t=0 and f=0 respectively. The main question you should ask yourself when using these functions is "where does t=0 (or f=0) live in my data?" and "Where do I want them to live?"

In general, fftshift should only be used after applying an fft/ifft. The output of these functions is given such that the first element represents f=0 and t=0 respectively. If you want to rearrange your data such that the f=0 and t=0 elements appear in the center, then fftshift is the right answer.

Without having a more thorough understanding of exactly what the data you are working with represents, it would be impossible to say whether any ifftshift or fftshift functions are necessary. Note that there are many situations in which one might use fft/fft2 and ifft/ifft2 correctly without ever needing to invoke fftshift or ifftshift.