dvPageText function

List<String> dvPageText(String source)

The prose in source, in order, without repeats.

Interpolated strings are skipped. 'Loaded at: $when' in a body is worse than nothing: it is visibly broken text on a page a crawler is reading.

Implementation

List<String> dvPageText(String source) {
  final found = <String>[];
  final seen = <String>{};

  // Directive lines removed first: a package URI is a string literal and is
  // not something anyone should read on the page.
  final body = source
      .split('\n')
      .where((String line) => !_directive.hasMatch(line))
      .join('\n');

  for (final RegExpMatch match in _literal.allMatches(body)) {
    final value = (match.group(1) ?? match.group(2) ?? '').trim();
    if (value.contains(r'$')) continue;
    if (!_isProse(value)) continue;
    if (seen.add(value)) found.add(value);
  }
  return found;
}