nextToken abstract method

Token? nextToken(
  1. IScanner scanner,
  2. ITokenizer? tokenizer
)

Gets the next token from the stream started from the character linked to this state.

  • scanner A textual string to be tokenized.
  • tokenizer A tokenizer class that controls the process. Returns The next token from the top of the stream.

Implementation

Token? nextToken(IScanner scanner, ITokenizer? tokenizer);