tokensByFormAll function

Map<String, List<UDToken>> tokensByFormAll(
  1. List<UDToken> tokens
)

Builds a map from surface form → all tokens with that form, in order.

Implementation

Map<String, List<UDToken>> tokensByFormAll(List<UDToken> tokens) {
  final map = <String, List<UDToken>>{};
  for (final t in tokens) {
    (map[t.form] ??= []).add(t);
    final lower = t.form.toLowerCase();
    if (lower != t.form) (map[lower] ??= []).add(t);
  }
  return map;
}