A bit more testing (using circular shots and nP_PolyOverlapsCircle) shows that the bounding check reckons that the circle and bounds are overlapping.
At that point the two are 14.1 pixels apart, well within the enemy (which is basically a rotating 45x45 square with some extrusions, and yes I'm rotating the polygon and it lines up quite nicely with the quad sprite if I draw the polygon using nP_DrawPoly).
But the next check (the actually poly check and the bit I don't understand) doesn't trigger.
At that point the two are 14.1 pixels apart, well within the enemy (which is basically a rotating 45x45 square with some extrusions, and yes I'm rotating the polygon and it lines up quite nicely with the quad sprite if I draw the polygon using nP_DrawPoly).
But the next check (the actually poly check and the bit I don't understand) doesn't trigger.