universal_automation_interface library
Pure-Dart contracts for the universal automation family.
This package defines the vocabulary shared by automation drivers and
frame pipelines: endpoints and session handles (oka-compatible), the
observe/act/verify driver contract, semantic snapshot types, fail-closed
typed specs, and structured events. It deliberately contains no protocol
clients and no lifecycle ownership — see the sibling packages
universal_browser_cdp, universal_browser_webdriver, and
universal_screencast.
Naming and semantics follow the oka session contract:
session-<name>-handle is the primary handle artifact and
session-<name>-<sub> are sub-handles. LeaseOwnership.borrowed marks
endpoints this package family must never stop.
Classes
- ActionRhythm
- Pre-action dwell: how long the input source rests before an action.
- AutomationAction
- Base class of driver actions.
- AutomationActionCatalog
-
Opt-in catalog of named actions a driver can invoke via
InvokeAction. - AutomationDriver
- The observe/act/verify contract every automation driver implements.
- AutomationEndpoint
- A resolved automation endpoint: where to attach and how to speak to it.
- AutomationEvent
- Structured, payload-free automation events.
- AxBounds
- Bounding box of a semantic node in logical pixels.
- AxNode
- One node of a semantic/accessibility tree.
- BehavioralDriver
- Drivers that can deliver actions under a declared behavior profile.
- BehaviorOutcome
- The typed result of a profiled dispatch: what was planned, what actually went out, and how the story ended.
- BehaviorPlan
-
The deterministic canonical plan
synthesizeproduces: steps inplannedAtUsorder, hashable, serializable, replayable byte-for-byte. - BehaviorProfile
- A declarative input-dynamics profile: how actions are delivered.
- BehaviorProjection
- The result of projectBehavior.
- BehaviorReceipts
- Receipt line encoders for behavior dispatches (ADR 0044).
- BehaviorReport
- The audit report: descriptive numbers about a plan and its profile.
- BehaviorRng
- Deterministic, version-pinned PRNG for behavior synthesis.
- BehaviorStep
- The timed input-step vocabulary: the one primitive language every behavior plan is written in, every transport lowers, and every audit consumes (ADR 0044).
- BezierPath
-
A single-arc curve: the control point is offset perpendicular to the
straight line by
curvature * distance, optionally overshooting the target along the direction of travel and correcting back. - CaptureOptions
- How one capture should be produced.
- CharStep
-
Deliver printable text as a character event (the lowering may choose
IME-style insertion, e.g. CDP
Input.insertText, for non-ASCII runes). - ClickAction
- Click an element located by CSS selector, or by role/name when the driver resolves roles through its semantic tree.
- ClickAtAction
-
Click at surface coordinates — the fallback tier for surfaces no
accessibility tree can see (canvas, games) and for pixel-grounded
agents (ADR 0053). Capability
pointerCoordinates; drivers without it refuse loudly. - ConfigurableCapture
- A driver that can honor CaptureOptions the base AutomationDriver.screenshot contract cannot express.
- ConsoleEntry
- One captured console line or page error from a live surface.
- ConsoleLogReader
- Reads buffered console output from a live surface.
- DirectPath
- Straight-line interpolation between path endpoints.
- DispatchedStep
- One dispatched step with its actual timing.
- DragAction
- Press at fromX/fromY, move to toX/toY, release — drag and drop, sliders, canvas gestures (ADR 0053). Behavioral profiles lower the whole path through humanized segments; plain dispatch is one press, one move, one release.
- DriverCapabilities
-
What a driver can do, declared up front so compositions can validate
before attaching (the
StorageCapabilitiespattern). - DwellStep
-
An explicit pause. Gaps in
plannedAtUsand explicit dwells carry the same semantics; explicit dwells make plans self-describing for audits. -
EffectReceipt<
T> - Transport completion and application effect are independent observations.
- ElementAction
-
Perform one action an element advertises —
AXConfirm,AXShowMenu,AXPick,AXIncrement, … (ADR 0064). - EvaluateAction
- Evaluate a read-only expression in the live surface.
- ExpressionEvaluator
- Evaluates an expression in the live surface and returns its value.
- FixedTiming
- A constant duration.
- FrameDelivered
- A frame was delivered to at least one sink. Carries size metadata, never the bytes themselves.
- InteractionSession
- A target-anchored manipulation lifecycle. It owns neither project writes nor permission: ending only stages a candidate for a separate adoption operation.
- InvokeAction
-
Invoke a named action the surface under test registered for automation
(the
invoketier — the dynamic-registry shape for drivers). - KeyDownStep
- Press a key: named key, optional platform keyCode and printable text payload.
- KeyPressAction
-
Press a named key (
Enter,Tab,Escape,Backspace,ArrowUp/Down/Left/Right, and the modifier namesShift/Control/Alt/Meta), optionally under modifiers — a chord likecontrol+T(ADR 0053). - KeystrokeCadence
- Keyboard delivery dynamics: inter-key (digraph) latency and key hold.
- KeyUpStep
- Release a key: named key, optional platform keyCode.
- MoveAction
- Move the pointer to (x, y) without pressing — hover affordances, tooltips, pre-positioning before a drag (ADR 0053).
- Navigate the surface to url.
- OsActivator
-
Brings an application to the front — the explicit focus handoff the
ADR 0064 ladder performs after a
foreground: trueopt-in. - OsAppCatalog
- Lists the running, anchorable applications (ADR 0004 facades — capability-granular OS surfaces so compositions cast to contracts, never to concrete drivers). Today macOS AX provides this; the Android AccessibilityService tier is the planned second source.
- OsAppRecord
- One running, anchorable OS application — the discovery record the app-catalog surface returns and every targeting call consumes (ADR 0004: the neutral shape behind the per-OS driver records).
- OsConsentRequester
- Requests the operating system's consent prompts for a tier's capabilities.
- OsDisplayCapture
-
Whole-display capture with a long-side cap; drivers whose base
screenshot()cannot resize declare this to opt intomaxPx. - OsWindowCapture
- Window discovery and window-scoped capture (needs the OS screen recording consent where applicable).
- OsWindowRecord
- One on-screen OS window — the record window-scoped capture consumes.
- PathModel
- How the pointer travels: the path shape over a move segment.
- PiecewiseTiming
-
An empirical distribution:
nequal-probability buckets delimited byn + 1strictly increasing boundaries; sampling is uniform bucket pick plus linear interpolation. - PipelineFailed
- Terminal pipeline failure, tagged with a coarse cause category so agents can classify without parsing details.
- PointerDownStep
-
Press a pointer button (
leftby default), optionally under keyboard modifiers — the step-level chord carrier (ADR 0053). - PointerMotion
- Pointer delivery dynamics: path shape, per-move duration, button hold, and the maximum stride between synthesized move points.
- PointerMoveStep
- Move the pointer to logical-pixel (x, y) over durationUs.
- PointerUpStep
- Release a pointer button, with the modifiers its press carried.
- ProfiledDriver
-
A
BehavioralDriverwrapper holding an immutable default profile. - ReactionDelay
- The observation→dispatch floor.
- ResolvedSession
- A resolved, live session: a driver over an attached endpoint.
- ScrollAction
-
Scroll the surface (or the scrollable containing
css) by distance logical pixels in direction. - SemanticTarget
- An exact semantic locator. Scopes are resolved uniquely, outermost first. A target is re-resolved on each operation; it is not a retained element ref.
- SessionBinding
-
One plan-or-ad-hoc session binding: a named transport pointing at a
live uri, or at a handle (
session-<name>-style artifact) the caller resolves via overrides. Borrowed sessions are detached from, never stopped. - SessionDescriptor
- Declarative description of an automation session.
- SessionHandles
- Session-handle naming convention shared with oka.
- SessionPacing
- Session-level pacing: noise-event rate and rhythm drift.
- SetValueAction
- Set an element's value directly — the semantic write path (ADR 0064).
- SinkDegraded
- A sink degraded or left the pipeline; remaining sinks are unaffected.
- Snapshot
- A semantic snapshot of the automated surface at a point in time.
- SourceStarted
- A frame source started emitting.
- SourceStopped
- A frame source stopped emitting. reason distinguishes requested stops from failures.
- StatSummary
- Descriptive statistics over a set of durations, in microseconds.
- SurfaceActionDescriptor
- Describes one named action the surface under test registered for automation — one entry of a driver's action catalog.
- TargetedAutomationDriver
- Optional capability; it does not add variants to the sealed action set.
- TimingDistribution
- A timing distribution over durations, in integer microseconds.
- TypeAction
- Focus an element and type text into it.
- TypedSpec
- Base class for fail-closed, typed composition specs.
- UniformTiming
-
A uniform duration over
[minMicros, maxMicros). - WheelStep
- Wheel/scroll deltas at the current pointer position.
Enums
- AutomationTransport
- Wire protocol family an automation endpoint speaks.
- BehaviorInterruptionCause
- Why a dispatch did not complete.
- BehaviorVerdict
- Terminal state of a profiled dispatch.
- CaptureFormat
- The image encoding a capture is delivered in.
- EffectDelivery
- Evidence about the one dispatch attempt.
- EffectObservation
- Independent application postcondition evidence.
- InteractionOrigin
- Physical input is observed; synthetic delivery and semantic invocation remain distinct even when they produce the same product operation.
- InteractionPhase
- LeaseOwnership
-
Ownership of a session's process lease, mirroring oka's
LeaseOwnership. - StartMode
- Whether a session is started by us or adopted from someone else.
Constants
-
keyboardModifiers
→ const Set<
String> - The four modifiers actions can carry.
- nominalTarget → const AxBounds
- Convenience re-export: the nominal target used by pre-flight projection when no real element bounds exist yet.
-
systemOpenUrlSchemes
→ const Set<
String> -
URI schemes a desktop driver will hand to the OS shell (ADR 0003).
Everything else — notably
javascript:anddata:— refuses loudly: the opener asks the operating system to handle the URL, and the allowlist keeps that surface inert.
Functions
-
auditBehavior(
BehaviorPlan plan, BehaviorProfile profile) → BehaviorReport -
Audits
planagainstprofile: self-consistency deltas, structural descriptives, and physical-plausibility flags. Pure; never throws on plan shape, and never returns a humanness verdict. -
canonicalHash(
Object? value) → String - SHA-256 of the canonical form, hex-encoded. Used for profile hashes, plan (stream) hashes, and receipt linkage.
-
canonicalJson(
Object? value) → String - Canonical JSON encoding for behavior artifacts (plans, profile hashes).
-
gridValue(
double pixels) → int - Quantizes logical pixels to the canonical hundredth-of-a-pixel grid.
-
modifierKeyName(
String modifier) → String -
The step-level key name for a modifier (the family's named-key
convention, same shape as
Enter). -
namedVirtualKeyCode(
String key) → int? -
Virtual key code for the family's named-key set (
Enter,Tab,Escape,Backspace, arrows, and the modifier names), ornulloutside it. Modifier codes follow the Windows VK conventions the web tiers use; OS tiers re-map onto their native codes. -
normalizeModifier(
String raw) → String? -
Normalizes one modifier alias (
ctrl,option,cmd,command,windows);nulloutside the vocabulary. -
openUrlViaSystem(
Uri url) → Future< void> -
Opens
urlthrough the platform's default handler (openon macOS,xdg-openon Linux,cmd /c starton Windows) — the desktop-tier lowering ofNavigateAction(ADR 0003). The opened browser is an untracked surface: the OS owns the process; this call never inspects or drives it. -
parseModifiers(
Iterable< Object?> raw) → List<String> - Parses a wire modifier list: aliases normalize, unknown names throw (fail-closed — a dropped modifier is a misfired chord).
-
pixelsFromGrid(
int grid) → double - Restores logical pixels from the canonical hundredth grid.
-
printableVirtualKeyCode(
String char) → int? -
Virtual key code for a printable ASCII character (uppercase letter,
digit, or punctuation), or
nulloutside that set. Shared by the behavior synthesis and by per-char key lowerings so both paths agree on codes. -
projectBehavior(
BehaviorProfile profile, int seed, AutomationAction action, {AxBounds? target, (double, double) start = (0, 0)}) → BehaviorProjection - Pre-flight projection: the plan plus its dispatch-time budget.
-
resolveSemanticTarget(
Snapshot snapshot, SemanticTarget target) → AxNode - Resolves one node in a fresh snapshot, preserving its adapter attributes.
-
sha256Hex(
List< int> bytes) → String -
SHA-256 of
bytes, lowercase hex. -
synthesizeBehavior(
BehaviorProfile profile, int seed, AutomationAction action, {AxBounds? target, (double, double) start = (0, 0)}) → BehaviorPlan - Deterministic plan synthesis (ADR 0044): same (profile, seed, action, target, start) always yields the same plan, byte-for-byte, on every platform.
-
validateOpenableUrl(
Uri url) → void - The scheme allowlist gate (ADR 0003), shared by every desktop driver's NavigateAction lowering: openUrlViaSystem applies it before spawning, and drivers apply it before their injected opener seam — an injected no-op must not bypass the guard.
-
verifyEffect<
T> ({required Future< void> dispatch(), required Future<T> observe(), required bool changed(T before, T after), bool isSatisfied(T value)?, Duration timeout = const Duration(seconds: 5), Duration dispatchTimeout = const Duration(seconds: 1), Duration pollInterval = const Duration(milliseconds: 50)}) → Future<EffectReceipt< T> > - Dispatches at most once, then observes with one monotonic deadline. A delivery timeout/error never causes replay. Later effect observation may still succeed. Futures must tolerate their completion after a timeout. Initial observation failure propagates: without a baseline no action runs.
Typedefs
-
DriverFactory
= Future<
ResolvedSession> Function(SessionBinding binding, Uri endpoint, Duration attachTimeout) - Builds a live session over one resolved endpoint — the extension seam of every session registry. Composition roots (the instrumented Flutter tier, a mobile self-widget-tree driver, a WebView tier) register factories without touching the toolkit.
Exceptions / Errors
- AutomationException
- Base class of all family exceptions: a stable message, a machine-readable kind, and structured details agents can classify without parsing prose.
- DriverUnsupportedException
- A driver was asked to perform something its capability set excludes.
- ElementNotFoundException
- A locator matched no node in the driver's latest observation.
- EndpointUnreachableException
- The endpoint did not answer or answered with an unexpected identity.
- ForegroundRequiredException
- A global-event verb was requested while the target application does not own the event stream (ADR 0064's focus contract).
- ProtocolException
- A protocol-level error response came back from the endpoint.
- SinkClosedException
- A frame sink was used after close — always a programming error.
- SpecViolationException
- A typed spec failed fail-closed validation before anything ran.
- TargetResolutionException
- No target or more than one target is a refusal, not an ordering rule.