Tokenizes a textual stream into a list of token structures.
scanner
List<Token> tokenizeStream(IScanner scanner);