GenericTokenizer class
Implements a default tokenizer class.
- Inheritance
-
- Object
- AbstractTokenizer
- GenericTokenizer
Constructors
Properties
- commentState ↔ ICommentState?
-
A token state to process comments.
getter/setter pairinherited
- decodeStrings ↔ bool
-
Decodes quoted strings.
getter/setter pairinherited
- hashCode → int
-
The hash code for this object.
no setterinherited
- lastTokenType ↔ TokenType
-
getter/setter pairinherited
- mergeWhitespaces ↔ bool
-
Merges whitespaces.
getter/setter pairinherited
- nextTokenValue ↔ Token?
-
getter/setter pairinherited
- numberState ↔ INumberState?
-
A token state to process numbers.
getter/setter pairinherited
- quoteState ↔ IQuoteState?
-
A token state to process quoted strings.
getter/setter pairinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
- scanner ↔ IScanner?
-
Gets the scanner
getter/setter pairinherited
- skipComments ↔ bool
-
Skips comments.
getter/setter pairinherited
- skipEof ↔ bool
-
Skips End-Of-File token at the end of stream.
getter/setter pairinherited
- skipUnknown ↔ bool
-
Skip unknown characters
getter/setter pairinherited
- skipWhitespaces ↔ bool
-
Skips whitespaces.
getter/setter pairinherited
- symbolState ↔ ISymbolState?
-
A token state to process symbols (single like "=" or muti-character like "<>")
getter/setter pairinherited
- unifyNumbers ↔ bool
-
Unifies numbers: "Integers" and "Floats" makes just "Numbers"
getter/setter pairinherited
- whitespaceState ↔ IWhitespaceState?
-
A token state to process white space delimiters.
getter/setter pairinherited
- wordState ↔ IWordState?
-
A token state to process words or indentificators.
getter/setter pairinherited
Methods
-
clearCharacterStates(
) → void -
Clears all character states.
inherited
-
getCharacterState(
int symbol) → ITokenizerState? -
Gest the state for a given character.
inherited
-
hasNextToken(
) → bool -
Finds out if the tokenizer has a next token.
Returns true if it has a next token, false otherwise.
inherited
-
nextToken(
) → Token? -
Gets the next token.
Returns next token
inherited
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
readNextToken(
) → Token? -
Reads the next token.
Returns next token
inherited
-
setCharacterState(
int fromSymbol, int toSymbol, ITokenizerState state) → void -
Sets the characters' state.
inherited
-
tokenizeBuffer(
String buffer) → List< Token> -
Provides a token for a string buffer.
inherited
-
tokenizeBufferToStrings(
String buffer) → List< String?> -
Creates a list of token values.
inherited
-
tokenizeStream(
IScanner scanner) → List< Token> -
Creates a list of tokens
inherited
-
tokenizeStreamToStrings(
IScanner scanner) → List< String?> -
Creates a list of token values.
inherited
-
toString(
) → String -
A string representation of this object.
inherited
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited