openvino_genai

On-device LLM inference for Flutter via Intel OpenVINO™ GenAI: streaming chat completion on CPU and GPU, with the load/fallback hardening a heterogeneous Windows fleet actually needs.

Unofficial. Not affiliated with, endorsed by, or supported by Intel Corporation. "OpenVINO" is a trademark of Intel Corporation or its subsidiaries.

Platform support

Platform	Status
Windows x64	✅ via openvino_genai_windows

What you get

• Streaming chat completion through the OpenVINO GenAI C API. Messages go to the native side as a chat history and the model's own chat template is applied there — no hand-formatted prompt markup.

• Device-ladder loading with a smoke test. Loads walk an ordered device ladder — default ['GPU', 'CPU']; set OpenVinoGenAiService.devicePreference = ['NPU', 'GPU', 'CPU'] for the full chain. ov_bridge_init can report success on machines whose accelerator then silently produces zero tokens; a tiny probe generation catches that, the failing device is struck out for the process lifetime, and the model transparently reloads on the next rung. Opt out with OpenVinoGenAiService.acceleratorSmokeTestEnabled = false. Devices that don't enumerate on the machine are skipped before any load is attempted (SR-IOV VMs, for example, expose no NPU at all).

  The NPU rung is experimental — validate before enabling. It needs the platform package built with OPENVINO_GENAI_BUNDLE_FULL=ON (the default bundle omits Intel's 72 MB NPU compiler), and OpenVINO's NPU plugin has been observed to fail-fast the whole process (no catchable error) on incompatible model/driver combinations even when the device is present. Only enable it for combinations you have actually seen work.

• Hardened lifecycle. Blocking native calls (model load/free) run in a background isolate; loads have a caller-set wait budget with a detached self-healing load behind it; generation timeouts measure stall (time since the last token), so slow-but-alive CPUs are never killed mid-answer.

• Typed failures (OpenVinoGenAiException hierarchy) instead of bare Exceptions, and doctor() for one-call diagnostics.

Quick start

dependencies:
  openvino_genai: ^0.1.0

import 'package:openvino_genai/openvino_genai.dart';

final llm = OpenVinoGenAiService();

// An OpenVINO IR model directory: openvino_model.xml/.bin + tokenizer files.
await llm.loadModel(r'C:\models\qwen2.5-1.5b-instruct-int4-ov');

await for (final token in llm.generateChatStream([
  const LlmMessage.system('You are a helpful assistant.'),
  const LlmMessage.user('Why is the sky blue?'),
])) {
  stdout.write(token);
}

Non-streaming: generateText(...). Cancel a running generation by cancelling the stream subscription, or via cancelGeneration().

Models

Any OpenVINO IR LLM export works — e.g. the pre-converted models in OpenVINO's Hugging Face collections, or your own export via optimum-intel. Point loadModel at the directory containing openvino_model.xml.

Sampling

llm.generateChatStream(
  messages,
  params: const LlmGenerationParams(maxTokens: 512, temperature: 0.4, topP: 0.9),
);

LlmGenerationParams.defaults is a balanced profile; .greedy is deterministic decoding. temperature: 0.0 always means greedy.

The rules of the road (0.x)

The native pipeline is a process-wide singleton: one model resident at a time, one generation at a time. Loading a different model swaps the first one out; a second concurrent generation throws LlmBusyException. A handle-based API is planned for 1.0.

One runtime pin per process. This package's platform implementation pins one exact OpenVINO GenAI runtime (see its README). Never ship a second copy of the OpenVINO runtime DLLs in the same app: Windows caches DLL modules by file name, so two copies at different versions silently cross-bind and fail in undebuggable ways.

Custom DLL locations

By default the bridge DLL and runtime are bundled next to your .exe by the Flutter build. If your installer relocates them:

OpenVinoBridgeBindings.dllPathOverride = r'C:\MyApp\engine\openvino_genai_bridge.dll';

The whole runtime DLL set must sit in that same directory.

Diagnostics

final report = await OpenVinoGenAiService().doctor();
print(report); // DLL path, bridge version, runtime pin, pipeline state

Route the package's log lines into your logger with openVinoGenAiLogger = (line) => ...;.

License

Apache-2.0. The OpenVINO runtime bundled by the platform package is © Intel Corporation, Apache-2.0 — see the openvino_genai_windows package for redistribution notices.