nextToken method

@override Token? nextToken(IScanner scanner, ITokenizer? tokenizer)

Ignore word (such as blanks and tabs), and return the tokenizer's next token.

• scanner A textual string to be tokenized.
• tokenizer A tokenizer class that controls the process. Returns The next token from the top of the stream.

Implementation

@override
Token? nextToken(IScanner scanner, ITokenizer? tokenizer) {
  var line = scanner.peekLine();
  var column = scanner.peekColumn();
  int nextSymbol;
  var tokenValue = '';
  for (nextSymbol = scanner.read();
      _map.lookup(nextSymbol) != null && _map.lookup(nextSymbol) != false;
      nextSymbol = scanner.read()) {
    tokenValue = tokenValue + String.fromCharCode(nextSymbol);
  }

  if (!CharValidator.isEof(nextSymbol)) {
    scanner.unread();
  }

  return Token(TokenType.Word, tokenValue, line, column);
}