have a separate list for each colour
Yeah, I would go with using seperate lists for each color, if you really want to go the list route.
If you're iterating through a list to find a 'red' item, having this in a loop is going to slow things down quite a bit, otherwise I guess it can't hurt.
You can take out the whole sorting process just by having seperate lists, as you will only require the first item in the list, instead of looking for the first item with X attribute in the list.