tokenizePageText function

List<PdfTextToken> tokenizePageText(PdfPageText page)

Splits a page's extracted text into word tokens (maximal runs of non-whitespace) with their page-space bounds.

Bounds come from PdfPageText.rectsFor, unioned across the runs a token spans, so a token broken across positioned runs still gets one covering box.

Implementation

List<PdfTextToken> tokenizePageText(PdfPageText page) {
  final tokens = <PdfTextToken>[];
  final text = page.text;
  final pattern = RegExp(r'\S+');
  for (final match in pattern.allMatches(text)) {
    final start = match.start;
    final end = match.end;
    tokens.add(PdfTextToken(
      text: text.substring(start, end),
      start: start,
      end: end,
      bounds: _union(page.rectsFor(start, end)),
    ));
  }
  return tokens;
}