tokenize method
Tokenizes the source
into a list list Token
s. Each Token
contains
information about where it appears in the source
and a TokenType.
Note that the list might be tokens that should be
Token.invisibleToParser, if you're passing them to a Parser
directly,
you need to filter them. When using the methods in this class, this will
be taken care of automatically.
Implementation
List<Token> tokenize(String source) {
final scanner =
Scanner(source, scanDriftTokens: options.useDriftExtensions);
final tokens = scanner.scanTokens();
return tokens;
}