I'm not really sure why scaling a terrain would cause any speedhit. The same amount of vertices are being processed no matter what the size. I suspect you're right about the other objects (especially if you're not combining them into single surface entities!) being the largest cause of the slowdown.