Actually I don't need a tokenizer for my project as it's kinda special. I just need fast syntax highlighting and I can assume there's space between each of my words :)
But yes, for true programming language highlighting it would be needed.
I can see the speed issues, but I have a couple of ideas I need to try out. I'll let you know how it goes :)
But yes, for true programming language highlighting it would be needed.
I can see the speed issues, but I have a couple of ideas I need to try out. I'll let you know how it goes :)