llm_replay_eval library
Deterministic record/replay ("cassettes") and evaluation for on-device LLM inference in Flutter.
HTTP VCRs can't help you here: on-device models never touch the network. llm_replay_eval records at the inference boundary instead — so your AI-feature tests run fast, offline, and byte-identical every time, even for in-process models.
Classes
- CaseResult
- The evaluated outcome of one EvalCase.
- Cassette
-
A named collection of recorded interactions — the in-memory representation
of one cassette file. Persistence lives in
CassetteStoreso this stays pure and unit-testable without dart:io. - CassetteStore
- Loads and saves Cassettes as JSON files on disk.
- ContainsText
- Passes when the output contains substring.
- EqualsText
- Passes when the output equals expected (optionally trimming whitespace).
- EvalCase
- One row of an eval dataset: a request, how to produce its output, and the checks the output must satisfy.
- EvalInput
- The thing an Evaluator judges: a model's output plus any metadata recorded alongside it (token counts, finish reason, etc.).
- EvalReport
- The aggregate result of running an EvalSuite.
- EvalResult
- The verdict of one Evaluator on one EvalInput.
- EvalSuite
- A dataset of EvalCases run against a ReplaySession.
- Evaluator
- Judges a model output against a single criterion.
- Interaction
- One recorded request→response pair.
- IsValidJson
- Passes when the output parses as JSON (object or array).
- JsonFieldEquals
-
Passes when a dotted path in the output JSON equals value.
Example path:
usage.tokensorchoices.0.text. - JsonHasKeys
- Passes when the output is a JSON object containing all of keys.
- JudgeVerdict
- A parsed judge verdict: a 0..1 score, a pass flag, and a reason.
- LlmJudge
- An Evaluator that asks a model to grade the output, with the judge call routed through a ReplaySession so the verdict is recorded once and replayed deterministically forever after — making LLM-as-judge evals hermetic and offline in CI, which is the whole point.
- MatchesPattern
- Passes when the output matches pattern anywhere.
- MaxOutputLength
- Passes when the output length is within maxChars characters.
- MaxTokens
-
Passes when
meta['tokens']is within maxTokens. A missing token count is treated as a failure so a budget can't silently pass un-instrumented runs. - RecordedChunk
- One streamed piece of a response, with the (optional) delay observed before it arrived. Recording the timing lets replay optionally reproduce the real cadence of on-device generation instead of dumping everything at once.
- RecordedResponse
- The recorded output of one inference call.
- ReplaySession
- Wraps real on-device inference calls with deterministic record/replay.
Enums
- ReplayMode
- Controls whether interactions are served from a cassette or recorded live.
- ResponseKind
- Whether a recorded response arrived all at once or token-by-token.
Functions
-
canonicalize(
Object? value) → Object? - Recursively sorts map keys so that two semantically-equal structures encode to identical JSON. Exposed for testing and for callers that want to store a canonical copy of the request alongside its fingerprint.
-
digestBytes(
List< int> bytes) → String - A short, stable digest of binary data, suitable for embedding inside a request map before fingerprinting.
-
fingerprintRequest(
Map< String, Object?> request) → String - Produces a stable, canonical fingerprint for an inference request.
-
parseJudgeVerdict(
String raw, {double passThreshold = 0.5}) → JudgeVerdict - Leniently parses a judge model's free-form reply into a JudgeVerdict.
-
replayModeFromString(
String? value, {ReplayMode fallback = ReplayMode.auto}) → ReplayMode -
Resolves the active ReplayMode from an environment value such as the
LLM_REPLAY_MODEdart-define / environment variable. -
satisfies(
Evaluator evaluator) → Matcher -
Adapts any synchronous Evaluator into a
flutter_testMatcher, so eval criteria can be used directly inexpect:
Exceptions / Errors
- CassetteFormatException
- Thrown when a cassette file cannot be understood (corrupt or too new).
- CassetteMissException
- Thrown in ReplayMode.replay when a request has no recorded interaction.