I have two 3D matrices. Each pixel is a gray scale value.
The first 3D matrix is a reference shape. e.g. imagine a 3D model of a mouth.
The second 3D matrix is an approximate subsection of the reference shape. e.g. imagine a tooth. The subsection is approximate as it is from a different mouth. Note the subsection could be slightly oblique.
The challenge is to identify what are the coordinates of the subsection inside the reference matrix. Expanding on the mouth example above, the challenge is to identify what tooth the second 3D matrix most accurately represents.
The output could be a 3D pixel location and an orientation value.
You can use imregister, imregtform and imwarp to automatically align two volumetric datasets. See matlab's documentation for an example...
Or if you want to work harder, convolve in 6D (convn) where you start with (x,y,z,theta=0,phi=0,psi=0) for the first matrix, and create a representation of the 3d tooth in all angle projections....
In general, you want to compare your subsection matrix B with every possible subsection matrix of the reference matrix A (so that it has the same size, as a matrix B) and find the most similar piece.
By comparison, I mean the estimation of the difference, e.g. if B_i is some subsection of matrix A, you can calculate:
d_i=sum(sum(sum((B_i-B).^2)));
After that, you need to find the minimum of $d_i$. If you are working with big matrices A(NxNxN) and B(MxMxM), there are (N-M)^3 possible submatrices B_i with the same size as matrix B. The algorithm will be quite expensive.
However, there is an alternative implementation using convolutions. You can take a look at the Particle Image Velocimetry algorithm, that is really similar to your problem.
