Round ViewPort... or something like this

Sounds like a job for stencil buffers. I know Shagwana did some stuff in Bmax with them. I found these two posts:
http://www.blitzbasic.com/Community/posts.php?topic=42622
http://blitzbasic.com/Community/posts.php?topic=49374