tokenizers/embedding library

Classes

TextAndTokens
A class that represents a text and its corresponding tokens.
TextAndVector
A class that represents a text and its corresponding embedding.