I'm trying to calculate the collision normal of two points. I need this for the collision response equations to calculate the new angular and linear velocities.
This scenario happens when, for example, the corners of two 2d or 3d boxes collide with each other. What would be their collision normal?
I now that in the case of a vertex and face collision the collision normal would be just the normal of the face.
It is undefined. Technically, the collision normal should be the normal at the surface of the their configuration space obstacle (ie the set of all transformations where the two objects intersect). Here is an interactive demo which may help:
http://www.pfirth.co.uk/minkowski.html
As a result, if the objects are not smooth, then a well defined normal will not exist at the points of non-differentiability on their boundary. Instead you get a cone of possible normals.
Thus, you could take any normal in that cone to be the one you use to separate the two objects, for example:
Now the principle of least action states that the correct solution to the motion here should be the normal which does minimal work, and the extrema here are just the walls of the cone. This means that in the absence of any other interactions, we could just assume that the impulse force is in the direction of one of the faces. To decide which one, again the action principle says that you should pick the normal which does least work. To figure this out, you just compute the collision response for both faces, then select the face whose impulse force has the least magnitude.