Suppose there are a number of convex polygons on a plane, perhaps a map. These polygons can bump up against each other and share an edge, but cannot overlap.
To test if two polygons P and Q overlap, first I can test each edge in P to see if it intersects with any of the edges in Q. If an intersection is found, I declare that P and Q intersect. If none intersect, I then have to test for the case that P is completely contained by Q, and vice versa. Next, there's the case that P==Q. Finally, there's the case that share a few edges, but not all of them. (These last two cases can probably be thought of as the same general case, but that might not be important.)
I have an algorithm that detects where two line segments intersect. If the two segments are co-linear, they are not considered to intersect for my purposes.
Have I properly enumerated the cases? Any suggestions for testing for these cases?
Note that I'm not looking to find the new convex polygon that is the intersection, I just want to know if an intersection exists. There are many well documented algorithms for finding the intersection, but I don't need to go through all the effort.
You could use this collision algorithm:
Your test cases should work, since you're checking the case where the polygons don't intersect at all (completely outside or completely inside), as well as where there is any form of partial intersection (edges intersect always if there is overlap).
For testing, I would just make sure to test every potential combination. The one missing above from what I see is a single edge shared, but one poly contained in the other. I would also add tests for some more complex poly shapes, from tri -> many sided, just as a precaution.
Also, if you had a U shaped poly that completely surrounded the poly, but didn't overlap, I believe your case would handle that, but I would add that as a check, as well.
Since altCognito already gave you a solution, I'll only point out an excellent book on computational geometry that might interest you.
There are several ways to detect intersection and / or containment between convex polygons. It all depends on how efficient you want the algorithm to be. Consider two convex polygons R and B with r and b vertices, respectively:
Here's an idea:
Find the center point of each polygon
Find the two points of each polygon closest to the center point of the other. They will be adjacent points in convex polygons. These define the nearest edge of each polygon, let's call the points {A, B} and {Y, Z}
Find the intersection of lines AB and YZ. If the line segments cross (the intersection on AB lies between A and B), your polygons intersect. If AB and XY are parallel ignore this condition, the next step will trap the problem.
There is one more case you need to check for, which is when the polygons intersect heavily enough that AB and XY are completely past each other and don't actually intersect. To trap this case, calculate the perpendicular distances of AB and XY to each polygons center points. If either center point is closer to the opposite polygon's line segment your polygon overlap heavily.
GJK collision detection should work.