AbstractTokenizer class abstract
Implements an abstract tokenizer class.
- Implemented types
- Implementers
Constructors
Properties
- commentState ↔ ICommentState?
-
A token state to process comments.
getter/setter pairoverride
- decodeStrings ↔ bool
-
Decodes quoted strings.
getter/setter pairoverride
- hashCode → int
-
The hash code for this object.
no setterinherited
- lastTokenType ↔ TokenType
-
getter/setter pair
- mergeWhitespaces ↔ bool
-
Merges whitespaces.
getter/setter pairoverride
- nextTokenValue ↔ Token?
-
getter/setter pair
- numberState ↔ INumberState?
-
A token state to process numbers.
getter/setter pairoverride
- quoteState ↔ IQuoteState?
-
A token state to process quoted strings.
getter/setter pairoverride
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
- scanner ↔ IScanner?
-
Gets the scanner
getter/setter pairoverride
- skipComments ↔ bool
-
Skips comments.
getter/setter pairoverride
- skipEof ↔ bool
-
Skips End-Of-File token at the end of stream.
getter/setter pairoverride
- skipUnknown ↔ bool
-
Skip unknown characters
getter/setter pairoverride
- skipWhitespaces ↔ bool
-
Skips whitespaces.
getter/setter pairoverride
- symbolState ↔ ISymbolState?
-
A token state to process symbols (single like "=" or muti-character like "<>")
getter/setter pairoverride
- unifyNumbers ↔ bool
-
Unifies numbers: "Integers" and "Floats" makes just "Numbers"
getter/setter pairoverride
- whitespaceState ↔ IWhitespaceState?
-
A token state to process white space delimiters.
getter/setter pairoverride
- wordState ↔ IWordState?
-
A token state to process words or indentificators.
getter/setter pairoverride
Methods
-
clearCharacterStates(
) → void - Clears all character states.
-
getCharacterState(
int symbol) → ITokenizerState? - Gest the state for a given character.
-
hasNextToken(
) → bool -
Finds out if the tokenizer has a next token.
Returns true if it has a next token, false otherwise.
override
-
nextToken(
) → Token? -
Gets the next token.
Returns next token
override
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
readNextToken(
) → Token? - Reads the next token. Returns next token
-
setCharacterState(
int fromSymbol, int toSymbol, ITokenizerState state) → void - Sets the characters' state.
-
tokenizeBuffer(
String buffer) → List< Token> -
Provides a token for a string buffer.
override
-
tokenizeBufferToStrings(
String buffer) → List< String?> -
Creates a list of token values.
override
-
tokenizeStream(
IScanner scanner) → List< Token> -
Creates a list of tokens
override
-
tokenizeStreamToStrings(
IScanner scanner) → List< String?> -
Creates a list of token values.
override
-
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited