Less the FPS less it will decay when it's on, as a example in my machine (p3-700, gforce1) this level runs at 80fps, and drops to 65fps when the googles are on.
But in my test machine (p2-400, tnt2)it runs at 30fps and go down to 25fps when on.
65/80 = 81% of full speed
25/30 = 83% of full speed
Seems pretty consistent to me.
Anyway it sounds like your framerates are well within acceptable levels in both cases (considering the hardware you're running it on)