neural/tokenizer/wordpiece_tokenizer library
Re-exports dart_bert_tokenizer for BERT WordPiece tokenization.
Used for code embedding with CodeBERT/DistilBERT models.
- anteater package
- documentation
- neural/tokenizer/wordpiece_tokenizer.dart
anteater package
- Libraries
- anteater
- api/anteater
- api/parallel_analyzer
- debt/cost_calculator
- debt/debt_aggregator
- debt/debt_config
- debt/debt_detector
- debt/debt_item
- debt/debt_report
- frontend/ir_generator
- frontend/kernel_reader
- frontend/source_loader
- ir/cfg/cfg_builder
- ir/cfg/control_flow_graph
- ir/ssa/ssa_builder
- metrics/complexity_calculator
- metrics/maintainability_index
- metrics/metrics_aggregator
- neural/cache/embedding_cache
- neural/inference/onnx_ffi
- neural/inference/onnx_runtime
- neural/tokenizer/wordpiece_tokenizer
- reasoner/abstract/abstract_domain
- reasoner/abstract/abstract_interpreter
- reasoner/abstract/bounds_checker
- reasoner/abstract/null_verifier
- reasoner/datalog/datalog_engine
- reasoner/datalog/fact_extractor
- reasoner/datalog/points_to_analysis
- rules/rule
- rules/rule_config
- rules/rule_registry
- rules/rule_runner
- rules/rules/quality/avoid_unnecessary_cast_rule
- rules/rules/quality/binary_expression_order_rule
- rules/rules/quality/prefer_async_await_rule
- rules/rules/quality/prefer_first_last_rule
- rules/rules/quality/prefer_trailing_comma_rule
- rules/rules/safety/avoid_dynamic_rule
- rules/rules/safety/avoid_global_state_rule
- rules/rules/safety/avoid_late_keyword_rule
- rules/rules/safety/no_empty_block_rule
- rules/rules/safety/no_equal_then_else_rule
- server/code_actions_provider
- server/diagnostics_provider
- server/hover_provider
- server/language_server
- version