StringTokenizer class

A tokenizer that breaks strings into lexical tokens for parsing.

StringTokenizer implements a lexer that scans source text and identifies tokens such as numbers, identifiers, operators, and punctuation. It maintains a current position and provides methods for consuming characters and patterns.

This is used by Parser to convert string expressions into parseable tokens before building unit objects.

Constructors

StringTokenizer(String source)

Creates a tokenizer for the entire source string.

StringTokenizer.fromRange(String source, int index, int endIndex)

Creates a tokenizer for a range within the source string.

Properties

endIndex → int

The end index (exclusive) for tokenization.

final

hashCode → int

The hash code for this object.

no setter, inherited

index ↔ int

The current position in the source string.

getter/setter pair

runtimeType → Type

A representation of the runtime type of the object.

no setter, inherited

source → String

The source string being tokenized.

final

Methods

eat(bool test(String char, int index), int? length) → ({int endIndex, int startIndex})?

Consumes characters that match a test function and returns their range.

eatBoolean() → bool?

Attempts to parse and consume a boolean value ('true' or 'false').

eatCharacter(String char) → bool

Attempts to consume a single char from the current position.

eatNumber() → double?

Attempts to parse and consume a number (integer or decimal) from the current position.

eatString(String string) → bool

Attempts to consume the exact string from the current position.

eatWhitespace() → bool

Consumes any whitespace characters from the current position.

noSuchMethod(Invocation invocation) → dynamic

Invoked when a nonexistent method or property is accessed.

inherited

tokenize() → List<Token>

Tokenizes the source string into a list of tokens.

toString() → String

A string representation of this object.

inherited

Operators

operator ==(Object other) → bool

The equality operator.

inherited