tokenizer/base_tokenizer library

Classes

BaseTokenizer
BaseTokenizer class for tokenization and detokenization.