EvalSuite class

Anthropic: a collection of tasks measuring specific capabilities or behaviors. Tasks in a suite typically share a broad goal.

Constructors

EvalSuite({required String name, required String agentName, required SuiteKind kind, required List<EvalTask> tasks, bool requireReferenceSolution = false, double taskPassThreshold = 1.0}): const

agentName → String: The agent these tasks are aimed at, e.g. card_agent / pkm_agent. Drives routing to the right AgentHarnessFactory and is the natural unit for filtering across multi-suite runs.
final
hashCode → int: The hash code for this object.
no setterinherited
kind → SuiteKind: final
name → String: final
requireReferenceSolution → bool: If true, every task must declare a referenceSolution. Strongly recommended for capability suites.
final
runtimeType → Type: A representation of the runtime type of the object.
no setterinherited
taskPassThreshold → double: Threshold for the report's task pass rate, in the range 0.0 .. 1.0. The default 1.0 preserves binary grader pass/fail decisions. Lower values accept partial credit when a completed trial's mean non-null score meets the threshold. Errored, timed-out and skipped trials never pass. Trial pass rate and pass@k / pass^k retain binary grader semantics.
final
tasks → List<EvalTask>: final

noSuchMethod(Invocation invocation) → dynamic: Invoked when a nonexistent method or property is accessed.
inherited
toString() → String: A string representation of this object.
inherited
validate() → List<String>: Validates the suite at construction time: ids unique, reference solutions present if required. Returns the list of problems (empty if valid).