but why then does it only occur when I turn to face in the direction were there is a high number polys.
Fillrate. If you look towards the wall that has nothing behind it and compare that to a situation where you have many walls behind the wall you're looking at, 3D-card still have to "draw" everything [whole model] in both cases but the 3D-card discards the polys that are behind the camera or have normals that face away from the camera just before rendering in the DX pipeline and doesn't actually draw them... but those walls-behind-the-walls it has to, and if they fill large part of the screen it slows down the framerate obviously.
You can try to control this FPS fluctuation if it's too bad for example by hiding and un-hiding blocks [with somekind of visibility code] from the level geometry - this is usually called as having "portals" or "BSP".