I've now noticed by hud sprites, which are parented to the camera, also shake but to a lesser degree.
The way to solve it is to use a second camera for all your HUD stuff.
You can also get this problem if your HUD sprites are too close to the camera, regardless of where in your 3D world everything is.