i wonder if its possable to speed it up?
I think so:
showTokens("this is a line of text, using a simple lexer to display each word.") Function showTokens(SLine$,identifier$=" ") Local m$, currentWord$ For loop = 1 To Len(SLine) m = Mid(SLine, loop, 1) If m = " " Then displayToken(currentWord) currentWord = "" Else currentWord = currentWord + m End If Next If currentWord <> "" Then displayToken(currentWord) End Function Function displayToken(tokenName$) Print tokenName End Function