I'm looking for an algorithm to detect if two rectangles intersect (one at an arbitrary angle, the other with only vertical/horizontal lines).
Testing if a corner of one is in the other ALMOST works. It fails if the rectangles form a cross-like shape.
It seems like a good idea to avoid using slopes of the lines, which would require special cases for vertical lines.
Here is what I think will take care of all possible cases. Do the following tests.
If the above 2 tests return false then these 2 rectangles do not overlap.
You could find the intersection of each side of the angled rectangle with each side of the axis-aligned one. Do this by finding the equation of the infinite line on which each side lies (i.e. v1 + t(v2-v1) and v'1 + t'(v'2-v'1) basically), finding the point at which the lines meet by solving for t when those two equations are equal (if they're parallel, you can test for that) and then testing whether that point lies on the line segment between the two vertices, i.e. is it true that 0 <= t <= 1 and 0 <= t' <= 1.
However, this doesn't cover the case when one rectangle completely covers the other. That you can cover by testing whether all four points of either rectangle lie inside the other rectangle.
m_pGladiator's answer is right and I prefer to it. Separating axis test is simplest and standard method to detect rectangle overlap. A line for which the projection intervals do not overlap we call a separating axis. Nils Pipenbrinck's solution is too general. It use dot product to check whether one shape is totally on the one side of the edge of the other. This solution is actually could induce to n-edge convex polygons. However, it is not optmized for two rectangles.
the critical point of m_pGladiator's answer is that we should check two rectangles' projection on both axises (x and y). If two projections are overlapped, then we could say these two rectangles are overlapped. So the comments above to m_pGladiator's answer are wrong.
for the simple situation, if two rectangles are not rotated, we present a rectangle with structure:
we name rectangle A, B with rectA, rectB.
if any one of the two rectangles are rotated, It may needs some efforts to determine the projection of them on x and y axises. Define struct RotatedRect as following:
the difference is how the width' is now a little different: widthA' for rectA:
Math.sqrt(rectA.width*rectA.width + rectA.height*rectA.height) * Math.cos(rectA.angle)
widthB' for rectB:Math.sqrt(rectB.width*rectB.width + rectB.height*rectB.height) * Math.cos(rectB.angle)
Could refer to a GDC(Game Development Conference 2007) PPT www.realtimecollisiondetection.net/pubs/GDC07_Ericson_Physics_Tutorial_SAT.ppt
This is what I would do, for the 3D version of this problem:
Model the 2 rectangles as planes described by equation P1 and P2, then write P1=P2 and derive from that the line of intersection equation, which won't exist if the planes are parallel (no intersection), or are in the same plane, in which case you get 0=0. In that case you will need to employ a 2D rectangle intersection algorithm.
Then I would see if that line, which is in the plane of both rectangles, passes through both rectangles. If it does, then you have an intersection of 2 rectangles, otherwise you don't (or shouldn't, I might have missed a corner case in my head).
To find if a line passes through a rectangle in the same plane, I would find the 2 points of intersection of the line and the sides of the rectangle (modelling them using line equations), and then make sure the points of intersections are with in range.
That is the mathematical descriptions, unfortunately I have no code to do the above.
I implemented it like this:
The matrix mB is any affine transform matrix that converts points in the B space to points in the A space. This includes simple rotation and translation, rotation plus scaling, and full affine warps, but not perspective warps.
It may not be as optimal as possible. Speed was not a huge concern. However it seems to work ok for me.
Enough has been said in other answers, so I'll just add pseudocode one-liner: