Lexer class
A lexer (also known as scanner or tokenizer) that breaks down Python subset
source code into a sequence of Token objects.
It recognizes keywords, identifiers, operators, literals, and
handles Python's significant whitespace to produce TokenType.INDENT and
TokenType.DEDENT tokens. It also ignores comments and handles newlines.
Properties
- current ↔ int
-
The current position (index) in the source string.
getter/setter pair
- hashCode → int
-
The hash code for this object.
no setterinherited
-
indentStack
↔ List<
int> -
A stack holding the indentation levels (number of leading spaces).
Used to detect
TokenType.INDENTandTokenType.DEDENT. Starts with 0.getter/setter pair - line ↔ int
-
The current line number in the source (1-based).
getter/setter pair
- lineStart ↔ int
-
The starting index of the current line in the source string, used for column calculation.
getter/setter pair
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
- source → String
-
The source code string being tokenized.
final
- start ↔ int
-
The starting position of the current lexeme being scanned in the source.
getter/setter pair
-
tokens
→ List<
Token> -
The list of tokens generated by the lexer. Populated by scanTokens.
final
Methods
-
addToken(
TokenType type, [Object? literal]) → void -
Creates a
Tokenwith the giventypeand optionalliteralvalue, using the current start, current, line, and calculated column. Adds the created token to the tokens list. -
advance(
) → String - Consumes the character at the current position and returns it, advancing the position.
-
currentColumn(
) → int -
frstring(
String quoteType, String prefix, [bool multiline = false]) → void - Scans an f-string or r-string literal. Finds closing quote, handles basic escapes in case of prefix 'f', but preserves content. fr-strings are not supported yet.
-
handleIndentation(
) → void - Checks and handles indentation changes at the beginning of a line.
-
identifier(
) → void -
Scans an identifier or a keyword.
Identifiers start with a letter or underscore, followed by letters, digits, or underscores.
If the scanned identifier matches a reserved word in keywords, the corresponding
keyword
TokenTypeis used; otherwise,TokenType.IDENTIFIERis used. -
isAlpha(
String c) → bool -
Checks if character
cis a letter (a-z, A-Z) or underscore (_). -
isAlphaNumeric(
String c) → bool -
Checks if character
cis a letter, digit, or underscore. -
isAtEnd(
) → bool - Returns true if the lexer has reached the end of the source string.
-
isBinaryDigit(
String c) → bool -
Checks if character
cis a binary digit (0 or 1). -
isDigit(
String c) → bool -
Checks if character
cis a digit (0-9). -
isHexDigit(
String c) → bool -
Checks if character
cis a hexadecimal digit (0-9, a-f, A-F). -
isOctalDigit(
String c) → bool -
Checks if character
cis an octal digit (0-7). -
match(
String expected) → bool -
Checks if the current character matches
expected. If so, consumes it and returns true. -
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
number(
) → void - Scans a number literal (integer or floating-point).
-
peek(
) → String - Returns the character at the current position without consuming it. Returns a null character ('\x00') if at the end of the source.
-
peekNext(
[int n = 1]) → String - Returns the character after the current position without consuming it. Returns a null character ('\x00') if near the end of the source.
-
scanToken(
) → void - Scans the next single token from the source code.
-
scanTokens(
) → List< Token> -
Scans the entire source string and returns a list of
Tokens. -
string(
String quoteType, [bool multiline = false]) → void -
Scans a string literal enclosed in single (') or double (") quotes
or multiline strings enclosed in ''' or """.
Handles unterminated strings by throwing a LexerError.
quoteTypeis the opening quote character (' or "). -
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited