Aw crap.
To decide which way the quads projected from the edges face, I calculated the normal of the quad (based on the first three vertex positions) and compared that to an average of the normal of the two triangles the quad's edge lies between. If the dotproduct was more than zero, I flipped the quad face. So all my volume edges are pointing on the same direction that the edge's triangles are pointing, and not in the opposite direction.
This works great for a convex outline. The outside of my shadow volume is perfect. However, it doesn't work with the smaller volume made out of the inside of the lip at the top. Although the edges form a complete loop, the sides of the volume point inward, and the bottom points up.
Is this indeed the problem?
How do I possibly figure out which direction to make the quads point? The only way I know how to is to make a list of all vertices for the loop, order them, calculate the angle change between each vertex, and add the changes up...or something like that. It sounds horribly complicated.
To decide which way the quads projected from the edges face, I calculated the normal of the quad (based on the first three vertex positions) and compared that to an average of the normal of the two triangles the quad's edge lies between. If the dotproduct was more than zero, I flipped the quad face. So all my volume edges are pointing on the same direction that the edge's triangles are pointing, and not in the opposite direction.
This works great for a convex outline. The outside of my shadow volume is perfect. However, it doesn't work with the smaller volume made out of the inside of the lip at the top. Although the edges form a complete loop, the sides of the volume point inward, and the bottom points up.
Is this indeed the problem?
How do I possibly figure out which direction to make the quads point? The only way I know how to is to make a list of all vertices for the loop, order them, calculate the angle change between each vertex, and add the changes up...or something like that. It sounds horribly complicated.