From what I understand, when you select a 'non-supported' refresh rate, from BlitzMax is supposed to pick a near match and apply some frame limiting to 'fit' your screen updates in there. (OpenGL bglSetSwapInterval, I presume)
The timing code is set up by Graphics and executed in Flip. bglSetSwapInterval does something different ( http://blitzwiki.org/index.php/BglSetSwapInterval )
Graphics will try and set the screen refresh to the desired rate. It will then perform a timing test to see if the refresh rate is OK. If it fails it just uses the default refresh rate and switches off vsync. In that case Flip will wait the required length of time before calling bglSwapBuffers.
It sounds like this is an issue in the timing test in Graphics.