I think I'm going to have to load each font multiple times, at different sizes, to minimise the amount of scaling.
That's what I do. I dislike scaled fonts due to the blurring. Works OK sometimes but only when scaling down. Mind you I use Bitmap fonts now made with Bitmap Font Generator (the old version gives better anti-aliasing in my opinion but I've asked the author to add the old method back in the new version as an option).