tokenizer_parser library
Tokenizer Parser public API.
This library exposes primitives to define literal and non-literal token models, compose grammars, and tokenize text or files into structured token trees.
Classes
- HasTokenizeLiteral
- Contract for models that tokenize raw input segments.
- HasTokenizeNonLiteral
- Contract for models that transform existing token streams.
- HasTokenizeStart
- Contract for models that can match at token-stream start.
- Input
- Represents a raw input segment with source coordinates.
- LiteralModel
- A regex-backed literal token model.
- NonLiteralModel
- A model that composes existing tokens into higher-level tokens.
- Token
- A token matched from input based on a TokenModel.
- TokenAlternatives
- A choice composition where the first matching alternative is selected.
- TokenBase
- Base abstraction for all token descriptors.
- Tokenizer
- Utility entry point for tokenizing input strings and files.
- TokenModel
- Base type for token models used by the tokenizer.
- TokenSequence
- A sequential composition where each element must match in order.