openvino_genai

On-device LLM and vision-language (VLM) inference for Flutter via Intel OpenVINO™ GenAI: streaming chat completion and image description on CPU and GPU, with the load/fallback hardening a heterogeneous Windows fleet actually needs.

Unofficial. Not affiliated with, endorsed by, or supported by Intel Corporation. "OpenVINO" is a trademark of Intel Corporation or its subsidiaries.

Platform support

Platform Status
Windows x64 ✅ via openvino_genai_windows

What you get

  • Streaming chat completion through the OpenVINO GenAI C API. Messages go to the native side as a chat history and the model's own chat template is applied there — no hand-formatted prompt markup.

  • Device-ladder loading with a smoke test. Loads walk an ordered device ladder — default ['GPU', 'CPU']; set OpenVinoGenAiService.devicePreference = ['NPU', 'GPU', 'CPU'] for the full chain. ov_bridge_init can report success on machines whose accelerator then silently produces zero tokens; a tiny probe generation catches that, the failing device is struck out for the process lifetime, and the model transparently reloads on the next rung. Opt out with OpenVinoGenAiService.acceleratorSmokeTestEnabled = false. Devices that don't enumerate on the machine are skipped before any load is attempted (SR-IOV VMs, for example, expose no NPU at all).

    The NPU rung is experimental — validate before enabling. It needs the platform package built with OPENVINO_GENAI_BUNDLE_FULL=ON (the default bundle omits Intel's 72 MB NPU compiler), and OpenVINO's NPU plugin has been observed to fail-fast the whole process (no catchable error) on incompatible model/driver combinations even when the device is present. Only enable it for combinations you have actually seen work.

  • Hardened lifecycle. Blocking native calls (model load/free) run in a background isolate; loads have a caller-set wait budget with a detached self-healing load behind it; generation timeouts measure stall (time since the last token), so slow-but-alive CPUs are never killed mid-answer.

  • Typed failures (OpenVinoGenAiException hierarchy) instead of bare Exceptions, and doctor() for one-call diagnostics.

Quick start

dependencies:
  openvino_genai: ^0.2.0

import 'package:openvino_genai/openvino_genai.dart';

final llm = OpenVinoGenAiService();

// An OpenVINO IR model directory: openvino_model.xml/.bin + tokenizer files.
await llm.loadModel(r'C:\models\qwen2.5-1.5b-instruct-int4-ov');

await for (final token in llm.generateChatStream([
  const LlmMessage.system('You are a helpful assistant.'),
  const LlmMessage.user('Why is the sky blue?'),
])) {
  stdout.write(token);
}

Non-streaming: generateText(...). Cancel a running generation by cancelling the stream subscription, or via cancelGeneration().

Models

Any OpenVINO IR LLM export works — e.g. the pre-converted models in OpenVINO's Hugging Face collections, or your own export via optimum-intel. Point loadModel at the directory containing openvino_model.xml.

Sampling

llm.generateChatStream(
  messages,
  params: const LlmGenerationParams(maxTokens: 512, temperature: 0.4, topP: 0.9),
);

LlmGenerationParams.defaults is a balanced profile; .greedy is deterministic decoding. temperature: 0.0 always means greedy.

Vision (VLM)

OpenVinoVlmService describes an image for a prompt through OpenVINO GenAI's VLM pipeline, on its own independent native lane (own DLL, lock, thread and wedge latch) — it can run concurrently with the LLM service on the one loaded runtime.

import 'package:image/image.dart' as img;   // your choice of decoder

final vlm = OpenVinoVlmService();
await vlm.loadModel(r'C:\models\internvl2-1b-int4-ov'); // an optimum-intel VLM export

// The package takes RGB8 pixels only: decode, convert and resize yourself.
final decoded = img.decodeImage(await File('photo.jpg').readAsBytes())!;
final resized = img.copyResize(decoded, width: 1024); // keep the longest edge <= ~1024
final image = VlmImage.rgb8(
  bytes: resized.getBytes(order: img.ChannelOrder.rgb),
  width: resized.width,
  height: resized.height,
);

final text = await vlm.describeImage(
  image,
  prompt: 'Describe this picture.',
  params: const LlmGenerationParams(maxTokens: 128, temperature: 0, topP: 1),
);

describeImageStream streams chunks; cancelling the subscription cancels the generation — but note cancellation is only observed at decode time: the vision encode and prompt prefill emit nothing and cannot be interrupted (10–30 s for a 640×480 frame on a desktop CPU, longer on slow machines). lastTimeToFirstToken / lastGenerateDuration let an app size its own budgets and "still looking" cues from measurements.

  • RGB8 contract. bytes.length == width * height * 3, interleaved R,G,B, no alpha, no row padding, edges 1..16384, ≤ 256 MiB — validated by VlmImage.rgb8 (throws VlmImageException). Camera sources are usually BGR: swap channels in your capture layer. No resize/normalisation happens in the package; the pipeline's own preprocessor tiles the image, so bigger images cost proportionally more encode time.
  • CPU-first. OpenVinoVlmService.devicePreference defaults to ['CPU']: an iGPU shared with the app's renderer can trip the Windows display watchdog during a long vision encode. ['GPU', 'CPU'] is opt-in for validated fleets, with the same enumeration guard and smoke test as the LLM lane; accelerator failures demote and throw (no automatic reload-retry — a VLM retry costs tens of seconds).
  • Unloading. unloadModel() cancels, waits for the thread on the Dart side, then frees natively with a bounded join. Size OpenVinoVlmService.unloadJoinBudget (default 180 s) from your measured describe durations and prefer letting a description finish before unloading; a thread that ignores cancellation for the whole budget latches the lane wedged for the process lifetime (VlmEngineWedgedException — advise a restart). The LLM lane is unaffected by a VLM wedge.

The rules of the road (0.x)

Each native pipeline is a process-wide singleton: one LLM and one VLM resident at a time, one generation per lane at a time. Loading a different model swaps the resident one out; a second concurrent generation on the same lane throws LlmBusyException / VlmBusyException. A handle-based API is planned for 1.0.

One runtime pin per process. This package's platform implementation pins one exact OpenVINO GenAI runtime (see its README). Never ship a second copy of the OpenVINO runtime DLLs in the same app: Windows caches DLL modules by file name, so two copies at different versions silently cross-bind and fail in undebuggable ways.

Custom DLL locations

By default the bridge DLL and runtime are bundled next to your .exe by the Flutter build. If your installer relocates them:

OpenVinoBridgeBindings.dllPathOverride = r'C:\MyApp\engine\openvino_genai_bridge.dll';
OpenVinoVlmBridgeBindings.dllPathOverride = r'C:\MyApp\engine\openvino_genai_vlm_bridge.dll';

Both bridge DLLs and the whole runtime DLL set must sit in that same directory — one copy of the runtime per process.

Diagnostics

final report = await OpenVinoGenAiService().doctor();
print(report); // DLL path, bridge version, runtime pin, pipeline state
print(await OpenVinoVlmService().doctor()); // same shape for the VLM lane

Route the package's log lines into your logger with openVinoGenAiLogger = (line) => ...;.

License

Apache-2.0. The OpenVINO runtime bundled by the platform package is © Intel Corporation, Apache-2.0 — see the openvino_genai_windows package for redistribution notices.

Libraries

openvino_genai
On-device LLM inference for Flutter via Intel OpenVINO GenAI.