potion8mTokenizer property

WordpieceTokenizer potion8mTokenizer
final

Tokenizer for Potion Base 8M.

For Potion 32M, explicitly import package:fonnx/tokenizers/potion_32m_tokenizer.dart and use its potion32mTokenizer. Keeping that vocabulary out of this library avoids loading it in debug builds of apps that only use 8M.

Implementation

static final potion8mTokenizer = WordpieceTokenizer(
  encoder: potionBase8mEncoder,
  decoder: potionBase8mDecoder,
  unkString: '[UNK]',
  unkToken: 1,
  startToken: 2,
  endToken: 3,
  maxInputTokens: 256,
  maxInputCharsPerWord: 100,
);