I also don't think that anyone can ever claim that a certain piece of code is pointless to be optimized under all circumstances, because you have no idea how many times it is being called or how important it is in the overall picture. This is for a very very time-critical highly-used lots-of-processing routine so little bits of difference really do add up.
I would be very surprised if I saw your code and could not find a more significant optimisation.
This kind of thing is so trivial it doesn't even matter on a Z80, let alone a 3Ghz Pentium 4.