which is faster using bytes or ints

Here is a tip - Don't optimise it or worry about fiddly stuff like this until and unless you know that there is a problem.

Write it in the simplest and most generic way first. Test it, then - if necessary - try out different optimisations.

It is a waste of time to over complicate something before you know that you need to.