SemanticCache class

Caches model responses by prompt similarity.

Prompts are embedded and stored with their responses in a DocumentStore; a lookup returns the response of the most similar prior prompt when its similarity reaches threshold. Use a cosine-metric collection (the default), so similarity is a cosine in [-1, 1].

Entries are scoped by namespace — use one per model and system prompt, since a cached answer is only valid for the setup that produced it — and expire after ttl.

Constructors

SemanticCache({required DocumentStore store, required Embedder embedder, String namespace = 'default', double threshold = 0.95, Duration? ttl})
Creates a cache.

Properties

embedder → Embedder
Embeds prompts.
final
hashCode → int
The hash code for this object.
no setterinherited
hits ↔ int
Lookups answered from the cache.
getter/setter pair
misses ↔ int
Lookups that missed.
getter/setter pair
namespace → String
Scope of the entries.
final
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
store → DocumentStore
Where entries live (may be shared with other data; entries are tagged).
final
threshold → double
Minimum cosine similarity for a hit.
final
ttl → Duration?
Entry lifetime, or null for no expiry.
final

Methods

clear() → Future<int>
Removes every entry in this namespace; returns how many.
lookup(String prompt) → Future<String?>
The cached response for a prompt similar to prompt, or null.
noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
put(String prompt, String response) → Future<void>
Stores response as the answer to prompt.
toString() → String
A string representation of this object.
inherited
wrap(ChatModel model) → ChatModel
model answering from this cache when it can.

Operators

operator ==(Object other) → bool
The equality operator.
inherited