ZeticMLange Flutter

Flutter SDK for running ZeticMLange on-device AI models on Android and iOS.

ZeticMLange loads models deployed from the ZETIC dashboard and runs inference through the native Android and iOS runtimes. The Flutter package exposes Dart APIs for general model inference, supported Hugging Face models, and LLM token generation.

Features

  • Run ZeticMLange models on-device from Flutter.
  • Load models by personal key, model name, and optional version.
  • Run tensor-based inference with Dart Tensor values.
  • Stream generated tokens from on-device LLM models.
  • Inject retrieved chunks or use a local on-device RAG pipeline with LLM models.
  • Select model mode, quantization, target, accelerator type, and cache policy.
  • Use the same Dart API surface on Android and iOS.

Platform support

Platform Minimum
Android API 24+
iOS iOS 16.6+
Dart 3.11.5+
Flutter 3.35.0+

Installation

Add the package to your Flutter app:

dependencies:
  zetic_mlange: ^1.9.0

Then install dependencies:

flutter pub get

This package requires the ZeticMLange native runtime to be available in the host app. Follow the Flutter setup guide for Android and iOS integration:

Basic inference

import 'dart:typed_data';

import 'package:zetic_mlange/zetic_mlange.dart';

Future<Float32List> runModel({
  required String personalKey,
  required String modelName,
  required Float32List inputValues,
}) async {
  final model = await ZeticMLangeModel.create(
    personalKey: personalKey,
    name: modelName,
  );

  try {
    final input = Tensor.float32List(
      inputValues,
      shape: const [1, 3, 224, 224],
    );

    final outputs = model.run([input]);
    return outputs.first.asFloat32List();
  } finally {
    model.close();
  }
}

LLM generation

import 'package:zetic_mlange/zetic_mlange.dart';

Future<void> generateText({
  required String personalKey,
  required String modelName,
}) async {
  final llm = await ZeticMLangeLLMModel.create(
    personalKey: personalKey,
    name: modelName,
    initOption: const LLMInitOption(nCtx: 4096),
  );

  try {
    llm.run('Explain on-device AI in one paragraph.');

    while (true) {
      final next = llm.waitForNextToken();
      if (next.isFinished) {
        break;
      }
      print(next.token);
    }

    llm.cleanUp();
  } finally {
    llm.close();
  }
}

Hugging Face models

Supported Hugging Face models can be loaded with ZeticMLangeHFModel:

final model = await ZeticMLangeHFModel.create(
  'owner/repository',
  userAccessToken: userAccessToken,
);

try {
  final outputs = model.run(inputs);
} finally {
  model.close();
}

See the Hugging Face model guide for supported formats and deployment requirements.

Lifecycle

ZeticMLangeModel, ZeticMLangeHFModel, and ZeticMLangeLLMModel expose close() and isClosed. Call close() when a model is no longer needed to release native runtime resources; repeated close() calls are safe no-ops. After isClosed is true, inference and token APIs throw MlangeException before invoking native FFI handles. ZeticMLangeLLMModel.deinit() remains as a deprecated compatibility alias for close().

RAG generation

External retrievers can pass ranked chunks to the native Android/iOS prompt assembler:

final result = await llm.runWithRetrievedChunks(
  query: 'What does the SDK support?',
  profile: const RagProfile.qwen25(),
  chunks: const [
    RetrievedChunk(text: 'The Flutter SDK supports local RAG bindings.'),
  ],
);

For end-to-end on-device retrieval, create a local RAG pipeline with decoder and embedder GGUF files, index documents, then run the LLM with that pipeline:

final pipeline = await LocalRagPipeline.create(
  profile: const RagProfile.qwen25(backboneGgufPath: 'decoder.gguf'),
  embedderGgufPath: 'embedder.gguf',
);

await pipeline.indexDocs(const [
  RagDocument(text: 'RAG injects retrieved context.', source: 'docs/rag.md'),
]);

await llm.runWithLocalRag(query: 'Explain RAG.', pipeline: pipeline);

Public API

The package exports:

  • ZeticMLangeModel
  • ZeticMLangeHFModel
  • ZeticMLangeLLMModel
  • RagProfile, RetrievedChunk, RagDocument, LocalRagPipeline
  • Tensor
  • DataType, Target, APType, ModelMode, QuantType
  • LLMModelMode, LLMTarget, LLMQuantType, LLMInitOption
  • LLMKVCacheCleanupPolicy, CacheHandlingPolicy
  • MlangeException

Documentation

Full documentation is available at docs.zetic.ai.

Useful starting points:

License

Apache-2.0. See LICENSE.

Development

FFI API Development Guide

To add or update FFI APIs exposed by the Android and iOS SDKs:

  1. Modify the FFI Contract: Update lib/src/ffi/mlange_ffi.h with your changes to the C-interface signatures.

  2. Generate Dart Bindings: Run ffigen to update lib/src/ffi/mlange_bindings.dart:

    dart run ffigen

    Note: You must have LLVM (Clang) installed locally.

  3. Update Dart FFI Logic: Modify lib/src/ffi/mlange_abi.dart to match the new signatures or structs.

  4. Local Development / Cross-Platform Sync: When developing Android or iOS SDKs alongside Flutter, you can sync/copy the local FFI header instead of fetching it from Nexus. Set the MLANGE_FLUTTER_DIR environment variable to point to your local mlange_flutter repository:

    export MLANGE_FLUTTER_DIR="/path/to/mlange_flutter"

    Then run ./fetch-deps.sh in the respective Android or iOS SDK repository.

Libraries

zetic_mlange