token/tokens library
Classes
- NewlineToken
- Tabs and empty spaces combined into one. NOT soft-wrap whitespace.
- TabsAndSpacesToken
- Tabs and empty spaces combined into one. NOT soft-wrap whitespace.
- TextPartToken
- WordPartToken
- Typically a syllable. But also a syllable preceed or followed by non-word characters like: "abc"
- WordToken