VirtualBox doesn't seem to like my XP driver disk though
Pretty much all emulators emulate your actual hardware with their own virtual devices, so rather than loading your true hardware drivers, you'll need to figure out what it pretends to have and load the drivers for that.