IncrementalTokenizer class
Incremental Tokenizer
Refers to tree-sitter's incremental tokenization strategy: When text is edited, only the region affected by the edit is re-tokenized; unmodified tokens before and after are reused directly (with offset shifts).
Algorithm:
- Determine the token range affected by the edit (dirty region) based on TextEdit.
- Retain tokens before the dirty region directly (offsets unchanged).
- Re-tokenize within the dirty region.
- Shift offsets and reuse tokens after the dirty region.
- Concatenate the three segments to form the new token list.
Convergence Condition (Key Optimization): During re-tokenization, once a newly generated token perfectly matches an old token (after shifting), stop immediately. This ensures the vast majority of edits only require re-tokenizing a small number of tokens.
Constructors
Properties
- hashCode → int
-
The hash code for this object.
no setterinherited
- runtimeType → Type
-
A representation of the runtime type of the object.
no setterinherited
Methods
-
appendText(
String text) → int - Appends text directly to the StringBuffer, avoiding O(n) copy from toString() + concatenation. Returns the complete text length after appending.
-
getCurrentText(
) → String - Gets the current text
-
getCurrentTokens(
) → List< LatexToken> - Gets the current token list (including EOF), using cache to avoid repeated creation
-
getTextLength(
) → int - Gets the current text length (avoids toString() allocation)
-
noSuchMethod(
Invocation invocation) → dynamic -
Invoked when a nonexistent method or property is accessed.
inherited
-
tokenize(
String text) → List< LatexToken> - Full initial tokenization
-
toString(
) → String -
A string representation of this object.
inherited
-
update(
String newText, TextEdit edit) → List< LatexToken> - Incremental update: Update the token list based on an edit operation
Operators
-
operator ==(
Object other) → bool -
The equality operator.
inherited