Completely the opposite, I'm afraid. x64 edition is very much faster than XP Pro on my machine. The UI is very snappy indeed and it tends to handle CPU and memory-intensive apps better but I haven't noticed any benefits on games that haven't been written with XP64 in mind (i.e. Far Cry with the 64-bit patch). Drivers aren't too much of an issue - anything that runs on Server 2003 will run on x64 and if the drivers aren't too low-level sometimes the 32-bit ones will work.
Confusingly my PC is almost exactly the same specs as your machine, Leiden: AMD64 3200+, 1GB PC3200 RAM, GeForce 6600GT. The motherboard's a nForce3 chipset running the 64bit drivers.
Confusingly my PC is almost exactly the same specs as your machine, Leiden: AMD64 3200+, 1GB PC3200 RAM, GeForce 6600GT. The motherboard's a nForce3 chipset running the 64bit drivers.