I've seen my mouse light up when the computer was turned off
Modern computers are never off, unless you kill the power before it hits your computer.
Power-on-LAN, power-on-modem, power-on-keyboard and power-on-date/time require the computer to be 'aware' of some of its peripherals at all times so they can boot up your computer if necesary.
The power button on an ATX case is not an old fashioned toggle-switch like the AT style cases used, but it simply signals the motherboard to do its thing and start booting.
You'll find that many motherboard have a green LED on the motherboard that will always be on to signal it has power even when the machine is turned 'off'.
Something to keep in mind while messing around with a machine exchanging peripherals while its still plugged into the wall -- unless you flip the master power swtich on the back of your case (if it even has one), there's a real danger you short out components.
the USB bus is another one that stays live while the computer is turned off -- you mentioned that you mouse sometimes kicks in. I have a bus-powered USB hub hooked up to my computer, and it has a LED that always glows blue, even if the machine is 'off'.
...Just imaging how much energy and resources are wasted in a year simply because hundreds of millions of devices never shut off and continue to use power, even though in 99% of the cases the end user would be just as happy (if not more so) if it would *truly* be off. Nothing like paying for power while you're not actually 'using' it.