nextToken method

@override Token? nextToken(IScanner scanner, ITokenizer? tokenizer)

Either delegate to a comment-handling state, or return a token with just a slash in it.

• scanner A textual string to be tokenized.
• tokenizer A tokenizer class that controls the process. Returns The next token from the top of the stream.

Implementation

@override
Token? nextToken(IScanner scanner, ITokenizer? tokenizer) {
  var line = scanner.peekLine();
  var column = scanner.peekColumn();
  var tokenValue = '';
  int nextSymbol;
  for (nextSymbol = scanner.read();
      !CharValidator.isEof(nextSymbol) &&
          nextSymbol != CR &&
          nextSymbol != LF;
      nextSymbol = scanner.read()) {
    tokenValue = tokenValue + String.fromCharCode(nextSymbol);
  }
  if (!CharValidator.isEof(nextSymbol)) {
    scanner.unread();
  }

  return Token(TokenType.Comment, tokenValue, line, column);
}