benchmarkTokenize static method

Future<BenchmarkResult> benchmarkTokenize(
  1. String text, {
  2. int iterations = 50,
})

Tokenization benchmark (Rust-powered)

Implementation

static Future<BenchmarkResult> benchmarkTokenize(
  String text, {
  int iterations = 50,
}) async {
  return benchmark(
    'Tokenize (${text.length} chars)',
    () async {
      tokenize(text: text);
    },
    iterations: iterations,
    category: BenchmarkCategory.rust,
  );
}