loadModel method

Future<LlamaBridgeSession> loadModel(String path, { int contextSize = 4096, int gpuLayers = 999, String? mmprojPath, int? imageTokenBudget, String? draftModelPath, int draftGpuLayers = 999, int maxDraftTokens = 3, int maxSequences = 1, })

Loads a GGUF model from an absolute filesystem path.

contextSize is the token context window. gpuLayers controls Metal offload; pass 0 to force CPU (e.g. the iOS Simulator).

Pass mmprojPath (an absolute path to a multimodal projector .gguf) to enable image input via LlamaBridgeSession.generate's images argument. imageTokenBudget then sets the vision encoder's per-image token budget (mtmd image_min_tokens/image_max_tokens) for vision models with dynamic resolution — Gemma supports 70, 140, 280, 560, or 1120, where higher budgets trade prefill time for fidelity (1120 suits OCR and small text). Null keeps the model's metadata default.

Pass draftModelPath (an absolute path to a drafter .gguf whose vocabulary matches the main model, e.g. a Gemma 4 MTP assistant) to enable speculative decoding: the drafter proposes up to maxDraftTokens tokens per step and the main model verifies them, leaving output identical but faster. draftGpuLayers controls the drafter's Metal offload independently of gpuLayers. Null keeps the session single-model. The drafter is best-effort: if it fails to load or its vocabulary is incompatible, the reason is logged natively and the session loads without speculation rather than failing.

Implementation

Future<LlamaBridgeSession> loadModel(
  String path, {
  int contextSize = 4096,
  int gpuLayers = 999,
  String? mmprojPath,
  int? imageTokenBudget,
  String? draftModelPath,
  int draftGpuLayers = 999,
  int maxDraftTokens = 3,
  int maxSequences = 1,
}) async {
  // Idempotent and safe under concurrent calls; see LlamaIsolate.start.
  await _isolate.start();
  _logger.logInformation(
    'Loading model $path (contextSize: $contextSize, '
    'gpuLayers: $gpuLayers, maxSequences: $maxSequences'
    '${mmprojPath == null ? '' : ', mmproj: $mmprojPath'}'
    '${draftModelPath == null ? '' : ', draft: $draftModelPath'}).',
  );
  final id = await _isolate.loadModel(
    ModelLoadRequest(
      modelPath: path,
      contextSize: contextSize,
      gpuLayers: gpuLayers,
      mmprojPath: mmprojPath,
      imageTokenBudget: imageTokenBudget,
      draftModel: draftModelPath == null
          ? null
          : DraftModelOptions(
              modelPath: draftModelPath,
              gpuLayers: draftGpuLayers,
              maxDraftTokens: maxDraftTokens,
            ),
      maxSequences: maxSequences,
    ),
  );
  return LlamaBridgeSession._(_isolate, id, maxSequences);
}