What about 3dGameStudio ?
I used it for two years, while it's possible to create a decent game with it, it'll take you about 5 years to do so (the one game that was decent for its time, Glider, has been in development for 5 years now and it still isn't released). There was another game, Josef's Tear, that switched engines due to limitations if I'm not mistaken -- it looked great but apparently it ran so poorly it was impossible to use 3DGS.
They have shaders in now but apparently the support for them is appalling. Physics are done with ODE, I've seen a few demos and nobody has done anything good with it so I'm hesitant to say that it's a proper implementation.
For the $200 ('Commercial') edition, you get to have 1 physics object in a level, 4 players in multiplayer (Wow! FOUR!? That's amazing! That's almost 60 less than Tribes 2! Boy gee golly gosh their engine is advanced!), you don't get render to texture, you don't get mirrors (which need render to texture or a mirror portal), and you are required to advertise their software on the game startup. If you want more, you have to shell out another $700.
So unless you plan on making very generic, ugly, slow, badly supported software I advise staying away from 3D GameStudio.