universal_automation_interface library

Pure-Dart contracts for the universal automation family.

This package defines the vocabulary shared by automation drivers and frame pipelines: endpoints and session handles (oka-compatible), the observe/act/verify driver contract, semantic snapshot types, fail-closed typed specs, and structured events. It deliberately contains no protocol clients and no lifecycle ownership — see the sibling packages universal_browser_cdp, universal_browser_webdriver, and universal_screencast.

Naming and semantics follow the oka session contract: session-<name>-handle is the primary handle artifact and session-<name>-<sub> are sub-handles. LeaseOwnership.borrowed marks endpoints this package family must never stop.

Classes

ActionRhythm
Pre-action dwell: how long the input source rests before an action.
AutomationAction
Base class of driver actions.
AutomationActionCatalog
Opt-in catalog of named actions a driver can invoke via InvokeAction.
AutomationDriver
The observe/act/verify contract every automation driver implements.
AutomationEndpoint
A resolved automation endpoint: where to attach and how to speak to it.
AutomationEvent
Structured, payload-free automation events.
AxBounds
Bounding box of a semantic node in logical pixels.
AxNode
One node of a semantic/accessibility tree.
BehavioralDriver
Drivers that can deliver actions under a declared behavior profile.
BehaviorOutcome
The typed result of a profiled dispatch: what was planned, what actually went out, and how the story ended.
BehaviorPlan
The deterministic canonical plan synthesize produces: steps in plannedAtUs order, hashable, serializable, replayable byte-for-byte.
BehaviorProfile
A declarative input-dynamics profile: how actions are delivered.
BehaviorProjection
The result of projectBehavior.
BehaviorReceipts
Receipt line encoders for behavior dispatches (ADR 0044).
BehaviorReport
The audit report: descriptive numbers about a plan and its profile.
BehaviorRng
Deterministic, version-pinned PRNG for behavior synthesis.
BehaviorStep
The timed input-step vocabulary: the one primitive language every behavior plan is written in, every transport lowers, and every audit consumes (ADR 0044).
BezierPath
A single-arc curve: the control point is offset perpendicular to the straight line by curvature * distance, optionally overshooting the target along the direction of travel and correcting back.
CaptureOptions
How one capture should be produced.
CharStep
Deliver printable text as a character event (the lowering may choose IME-style insertion, e.g. CDP Input.insertText, for non-ASCII runes).
ClickAction
Click an element located by CSS selector, or by role/name when the driver resolves roles through its semantic tree.
ClickAtAction
Click at surface coordinates — the fallback tier for surfaces no accessibility tree can see (canvas, games) and for pixel-grounded agents (ADR 0053). Capability pointerCoordinates; drivers without it refuse loudly.
ConfigurableCapture
A driver that can honor CaptureOptions the base AutomationDriver.screenshot contract cannot express.
ConsoleEntry
One captured console line or page error from a live surface.
ConsoleLogReader
Reads buffered console output from a live surface.
DirectPath
Straight-line interpolation between path endpoints.
DispatchedStep
One dispatched step with its actual timing.
DragAction
Press at fromX/fromY, move to toX/toY, release — drag and drop, sliders, canvas gestures (ADR 0053). Behavioral profiles lower the whole path through humanized segments; plain dispatch is one press, one move, one release.
DriverCapabilities
What a driver can do, declared up front so compositions can validate before attaching (the StorageCapabilities pattern).
DwellStep
An explicit pause. Gaps in plannedAtUs and explicit dwells carry the same semantics; explicit dwells make plans self-describing for audits.
EffectReceipt<T>
Transport completion and application effect are independent observations.
ElementAction
Perform one action an element advertises — AXConfirm, AXShowMenu, AXPick, AXIncrement, … (ADR 0064).
EvaluateAction
Evaluate a read-only expression in the live surface.
ExpressionEvaluator
Evaluates an expression in the live surface and returns its value.
FixedTiming
A constant duration.
FrameDelivered
A frame was delivered to at least one sink. Carries size metadata, never the bytes themselves.
InteractionSession
A target-anchored manipulation lifecycle. It owns neither project writes nor permission: ending only stages a candidate for a separate adoption operation.
InvokeAction
Invoke a named action the surface under test registered for automation (the invoke tier — the dynamic-registry shape for drivers).
KeyDownStep
Press a key: named key, optional platform keyCode and printable text payload.
KeyPressAction
Press a named key (Enter, Tab, Escape, Backspace, ArrowUp/Down/Left/Right, and the modifier names Shift/Control/Alt/Meta), optionally under modifiers — a chord like control+T (ADR 0053).
KeystrokeCadence
Keyboard delivery dynamics: inter-key (digraph) latency and key hold.
KeyUpStep
Release a key: named key, optional platform keyCode.
MoveAction
Move the pointer to (x, y) without pressing — hover affordances, tooltips, pre-positioning before a drag (ADR 0053).
Navigate the surface to url.
OsActivator
Brings an application to the front — the explicit focus handoff the ADR 0064 ladder performs after a foreground: true opt-in.
OsAppCatalog
Lists the running, anchorable applications (ADR 0004 facades — capability-granular OS surfaces so compositions cast to contracts, never to concrete drivers). Today macOS AX provides this; the Android AccessibilityService tier is the planned second source.
OsAppRecord
One running, anchorable OS application — the discovery record the app-catalog surface returns and every targeting call consumes (ADR 0004: the neutral shape behind the per-OS driver records).
OsConsentRequester
Requests the operating system's consent prompts for a tier's capabilities.
OsDisplayCapture
Whole-display capture with a long-side cap; drivers whose base screenshot() cannot resize declare this to opt into maxPx.
OsWindowCapture
Window discovery and window-scoped capture (needs the OS screen recording consent where applicable).
OsWindowRecord
One on-screen OS window — the record window-scoped capture consumes.
PathModel
How the pointer travels: the path shape over a move segment.
PiecewiseTiming
An empirical distribution: n equal-probability buckets delimited by n + 1 strictly increasing boundaries; sampling is uniform bucket pick plus linear interpolation.
PipelineFailed
Terminal pipeline failure, tagged with a coarse cause category so agents can classify without parsing details.
PointerDownStep
Press a pointer button (left by default), optionally under keyboard modifiers — the step-level chord carrier (ADR 0053).
PointerMotion
Pointer delivery dynamics: path shape, per-move duration, button hold, and the maximum stride between synthesized move points.
PointerMoveStep
Move the pointer to logical-pixel (x, y) over durationUs.
PointerUpStep
Release a pointer button, with the modifiers its press carried.
ProfiledDriver
A BehavioralDriver wrapper holding an immutable default profile.
ReactionDelay
The observation→dispatch floor.
ResolvedSession
A resolved, live session: a driver over an attached endpoint.
ScrollAction
Scroll the surface (or the scrollable containing css) by distance logical pixels in direction.
SemanticTarget
An exact semantic locator. Scopes are resolved uniquely, outermost first. A target is re-resolved on each operation; it is not a retained element ref.
SessionBinding
One plan-or-ad-hoc session binding: a named transport pointing at a live uri, or at a handle (session-<name>-style artifact) the caller resolves via overrides. Borrowed sessions are detached from, never stopped.
SessionDescriptor
Declarative description of an automation session.
SessionHandles
Session-handle naming convention shared with oka.
SessionPacing
Session-level pacing: noise-event rate and rhythm drift.
SetValueAction
Set an element's value directly — the semantic write path (ADR 0064).
SinkDegraded
A sink degraded or left the pipeline; remaining sinks are unaffected.
Snapshot
A semantic snapshot of the automated surface at a point in time.
SourceStarted
A frame source started emitting.
SourceStopped
A frame source stopped emitting. reason distinguishes requested stops from failures.
StatSummary
Descriptive statistics over a set of durations, in microseconds.
SurfaceActionDescriptor
Describes one named action the surface under test registered for automation — one entry of a driver's action catalog.
TargetedAutomationDriver
Optional capability; it does not add variants to the sealed action set.
TimingDistribution
A timing distribution over durations, in integer microseconds.
TypeAction
Focus an element and type text into it.
TypedSpec
Base class for fail-closed, typed composition specs.
UniformTiming
A uniform duration over [minMicros, maxMicros).
WheelStep
Wheel/scroll deltas at the current pointer position.

Enums

AutomationTransport
Wire protocol family an automation endpoint speaks.
BehaviorInterruptionCause
Why a dispatch did not complete.
BehaviorVerdict
Terminal state of a profiled dispatch.
CaptureFormat
The image encoding a capture is delivered in.
EffectDelivery
Evidence about the one dispatch attempt.
EffectObservation
Independent application postcondition evidence.
InteractionOrigin
Physical input is observed; synthetic delivery and semantic invocation remain distinct even when they produce the same product operation.
InteractionPhase
LeaseOwnership
Ownership of a session's process lease, mirroring oka's LeaseOwnership.
StartMode
Whether a session is started by us or adopted from someone else.

Constants

keyboardModifiers → const Set<String>
The four modifiers actions can carry.
nominalTarget → const AxBounds
Convenience re-export: the nominal target used by pre-flight projection when no real element bounds exist yet.
systemOpenUrlSchemes → const Set<String>
URI schemes a desktop driver will hand to the OS shell (ADR 0003). Everything else — notably javascript: and data: — refuses loudly: the opener asks the operating system to handle the URL, and the allowlist keeps that surface inert.

Functions

auditBehavior(BehaviorPlan plan, BehaviorProfile profile) → BehaviorReport
Audits plan against profile: self-consistency deltas, structural descriptives, and physical-plausibility flags. Pure; never throws on plan shape, and never returns a humanness verdict.
canonicalHash(Object? value) → String
SHA-256 of the canonical form, hex-encoded. Used for profile hashes, plan (stream) hashes, and receipt linkage.
canonicalJson(Object? value) → String
Canonical JSON encoding for behavior artifacts (plans, profile hashes).
gridValue(double pixels) → int
Quantizes logical pixels to the canonical hundredth-of-a-pixel grid.
modifierKeyName(String modifier) → String
The step-level key name for a modifier (the family's named-key convention, same shape as Enter).
namedVirtualKeyCode(String key) → int?
Virtual key code for the family's named-key set (Enter, Tab, Escape, Backspace, arrows, and the modifier names), or null outside it. Modifier codes follow the Windows VK conventions the web tiers use; OS tiers re-map onto their native codes.
normalizeModifier(String raw) → String?
Normalizes one modifier alias (ctrl, option, cmd, command, windows); null outside the vocabulary.
openUrlViaSystem(Uri url) → Future<void>
Opens url through the platform's default handler (open on macOS, xdg-open on Linux, cmd /c start on Windows) — the desktop-tier lowering of NavigateAction (ADR 0003). The opened browser is an untracked surface: the OS owns the process; this call never inspects or drives it.
parseModifiers(Iterable<Object?> raw) → List<String>
Parses a wire modifier list: aliases normalize, unknown names throw (fail-closed — a dropped modifier is a misfired chord).
pixelsFromGrid(int grid) → double
Restores logical pixels from the canonical hundredth grid.
printableVirtualKeyCode(String char) → int?
Virtual key code for a printable ASCII character (uppercase letter, digit, or punctuation), or null outside that set. Shared by the behavior synthesis and by per-char key lowerings so both paths agree on codes.
projectBehavior(BehaviorProfile profile, int seed, AutomationAction action, {AxBounds? target, (double, double) start = (0, 0)}) → BehaviorProjection
Pre-flight projection: the plan plus its dispatch-time budget.
resolveSemanticTarget(Snapshot snapshot, SemanticTarget target) → AxNode
Resolves one node in a fresh snapshot, preserving its adapter attributes.
sha256Hex(List<int> bytes) → String
SHA-256 of bytes, lowercase hex.
synthesizeBehavior(BehaviorProfile profile, int seed, AutomationAction action, {AxBounds? target, (double, double) start = (0, 0)}) → BehaviorPlan
Deterministic plan synthesis (ADR 0044): same (profile, seed, action, target, start) always yields the same plan, byte-for-byte, on every platform.
validateOpenableUrl(Uri url) → void
The scheme allowlist gate (ADR 0003), shared by every desktop driver's NavigateAction lowering: openUrlViaSystem applies it before spawning, and drivers apply it before their injected opener seam — an injected no-op must not bypass the guard.
verifyEffect<T>({required Future<void> dispatch(), required Future<T> observe(), required bool changed(T before, T after), bool isSatisfied(T value)?, Duration timeout = const Duration(seconds: 5), Duration dispatchTimeout = const Duration(seconds: 1), Duration pollInterval = const Duration(milliseconds: 50)}) → Future<EffectReceipt<T>>
Dispatches at most once, then observes with one monotonic deadline. A delivery timeout/error never causes replay. Later effect observation may still succeed. Futures must tolerate their completion after a timeout. Initial observation failure propagates: without a baseline no action runs.

Typedefs

DriverFactory = Future<ResolvedSession> Function(SessionBinding binding, Uri endpoint, Duration attachTimeout)
Builds a live session over one resolved endpoint — the extension seam of every session registry. Composition roots (the instrumented Flutter tier, a mobile self-widget-tree driver, a WebView tier) register factories without touching the toolkit.

Exceptions / Errors

AutomationException
Base class of all family exceptions: a stable message, a machine-readable kind, and structured details agents can classify without parsing prose.
DriverUnsupportedException
A driver was asked to perform something its capability set excludes.
ElementNotFoundException
A locator matched no node in the driver's latest observation.
EndpointUnreachableException
The endpoint did not answer or answered with an unexpected identity.
ForegroundRequiredException
A global-event verb was requested while the target application does not own the event stream (ADR 0064's focus contract).
ProtocolException
A protocol-level error response came back from the endpoint.
SinkClosedException
A frame sink was used after close — always a programming error.
SpecViolationException
A typed spec failed fail-closed validation before anything ran.
TargetResolutionException
No target or more than one target is a refusal, not an ordering rule.