display caps

But I have to look into the flags to see if something indicates that it's out of the monitor's range.

Take a look at ChangeDisplaySettings. Particularly the CDS_TEST (tests if the requested graphics mode could be set) flag. May be what you need ...