deduplicateMemories method

List<MemoryCandidate> deduplicateMemories(List<MemoryCandidate> candidates, List<String> existing)

Remove candidates whose content is already present in existing.

Implementation

List<MemoryCandidate> deduplicateMemories(
  List<MemoryCandidate> candidates,
  List<String> existing,
) {
  final normalised = existing.map((e) => _normalise(e)).toSet();

  return candidates.where((c) {
    final norm = _normalise(c.content);
    // Exact match or high-overlap substring.
    if (normalised.contains(norm)) return false;
    return !normalised.any((e) => _similarity(e, norm) > 0.85);
  }).toList();
}