ITokenizer class abstract

A tokenizer divides a string into tokens.

This class is highly customizable with regard to exactly how this division occurs, but it also has defaults that are suitable for many languages. This class assumes that the character values read from the string lie in the range 0-255. For example, the Unicode value of a capital A is 65, so String.fromCharCode(65) prints out a capital A.

The behavior of a tokenizer depends on its character state table. This table is an array of 256 TokenizerState states. The state table decides which state to enter upon reading a character from the input string.

For example, by default, upon reading an 'A', a tokenizer will enter a "word" state. This means the tokenizer will ask a IWordState object to consume the 'A', along with the characters after the 'A' that form a word. The state's responsibility is to consume characters and return a complete token.

The default table sets a SymbolState for every character from 0 to 255, and then overrides this with:

.. list-table:: :header-rows: 1 :widths: 10, 10, 25

* - From
  - To
  - State

* - 0
  - ' '
  - whitespaceState
* - 'a',
  - 'z'
  - wordState
* - 'A'
  - 'Z'
  - wordState
* - 160
  - 255
  - wordState
* - '0'
  - '9'
  - numberState
* - '-'
  - '-'
  - numberState
* - '.'
  - '.'
  - numberState
* - '"'
  - '"'
  - quoteState
* - '\''
  - '\''
  - quoteState
* - '/'
  - '/'
  - slashState

In addition to allowing modification of the state table, this class makes each of the states above available. Some of these states are customizable. For example, wordState allows customization of what characters can be part of a word, after the first character.

Implementers

Constructors

ITokenizer()

Properties

commentState ↔ ICommentState?
A token state to process comments.
getter/setter pair
decodeStrings ↔ bool
Decodes quoted strings.
getter/setter pair
hashCode → int
The hash code for this object.
no setterinherited
mergeWhitespaces ↔ bool
Merges whitespaces.
getter/setter pair
numberState ↔ INumberState?
A token state to process numbers.
getter/setter pair
quoteState ↔ IQuoteState?
A token state to process quoted strings.
getter/setter pair
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
scanner ↔ IScanner?
The stream scanner to tokenize.
getter/setter pair
skipComments ↔ bool
Skips comments.
getter/setter pair
skipEof ↔ bool
Skips End-Of-File token at the end of stream.
getter/setter pair
skipUnknown ↔ bool
Skip unknown characters
getter/setter pair
skipWhitespaces ↔ bool
Skips whitespaces.
getter/setter pair
symbolState ↔ ISymbolState?
A token state to process symbols (single like "=" or muti-character like "<>")
getter/setter pair
unifyNumbers ↔ bool
Unifies numbers: "Integers" and "Floats" makes just "Numbers"
getter/setter pair
whitespaceState ↔ IWhitespaceState?
A token state to process white space delimiters.
getter/setter pair
wordState ↔ IWordState?
A token state to process words or indentificators.
getter/setter pair

Methods

hasNextToken() → bool
Checks if there is the next token exist. Returns true if scanner has the next token.
nextToken() → Token?
Gets the next token from the scanner. Returns Next token of null if there are no more tokens left.
noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
tokenizeBuffer(String buffer) → List<Token>
Tokenizes a string buffer into a list of tokens structures.
tokenizeBufferToStrings(String buffer) → List<String?>
Tokenizes a string buffer into a list of strings.
tokenizeStream(IScanner scanner) → List<Token>
Tokenizes a textual stream into a list of token structures.
tokenizeStreamToStrings(IScanner scanner) → List<String?>
Tokenizes a textual stream into a list of strings.
toString() → String
A string representation of this object.
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited