tokenizeStream abstract method

List<Token> tokenizeStream(
  1. IScanner scanner
)

Tokenizes a textual stream into a list of token structures.

  • scanner A textual stream to be tokenized. Returns A list of token structures.

Implementation

List<Token> tokenizeStream(IScanner scanner);