I think we might need a bit more info to give an answer for this. I use Sswift's system and I have not seem that sort of performance hit.
What resolution are you making the shadow maps? Are you using soft shadows? Soft shadows can kill things quickly. How large are the meshes involved? (poly count?)
What resolution are you making the shadow maps? Are you using soft shadows? Soft shadows can kill things quickly. How large are the meshes involved? (poly count?)