tokenizer_moqasem library
Tokenizer Moqasem - Intelligent tokenization with Arabic support
A comprehensive tokenization plugin for Flutter applications with advanced Arabic language support, fuzzy matching, and AI-powered text analysis.
Built with 95% AI assistance using Claude (Anthropic).
Features:
- Multiple tokenization strategies
- Advanced Arabic text processing
- Fuzzy matching and similarity metrics
- Language detection
- Phonetic matching
- Performance optimizations
Classes
- ArabicProcessor
- Handles advanced Arabic text processing
- ArabicStemmer
- Handles Arabic word stemming and root extraction
- PhoneticMatch
- Phonetic match result
- PhoneticMatcher
- Handles phonetic matching for Arabic words
- SimilarityCalculator
- Handles various similarity calculations
- SimilarityResult
- Class to hold similarity results
- TextStatistics
- Class to hold text statistics
- Tokenizer
- Main Tokenizer class with comprehensive text processing capabilities
- TokenizerConfig
- Configuration class for tokenizer
- TokenizerConstants
- Constants used throughout the tokenizer
- WordSimilarity
- Class to hold word similarity information
Enums
- Language
- Enum for language detection
- TokenizationType
- Enum for different tokenization strategies