Offline SLM 🧠⚡

A fast, privacy-first Flutter package for running Small Language Models (SLMs) completely offline on-device with ready-to-use drop-in AI chatbot widgets.

📱 Runnable Demo App

Looking for a full, runnable reference application? Check out the official offline_SLM_app repository featuring:

Instant Simulator Mode preview for fast UI testing with zero waiting.
Model catalog switcher with on-the-fly streaming GGUF downloads.
Real-time token per second (TPS) and latency metrics.
Dark / Light Material 3 theme toggle.

🌟 Why Offline SLM?

🔒 100% On-Device Privacy: No prompts, chats, or personal data ever leave the user's phone.
⚡ Zero Cloud Latency & Zero API Bills: No expensive OpenAI/Anthropic/Gemini subscription or token costs.
✈️ Works Anywhere: Full AI chat capabilities in airplane mode, remote areas, or strictly offline environments.
🎨 Drop-in UI & Headless Support: Use the complete conversational OfflineChatWidget or build your own custom interface with OfflineChatController.
🧩 Multi-Architecture Prompt Templating: Automated prompt formatting and stop-sequence injection for ChatML (SmolLM2, Qwen 2.5), Llama 3.2, Gemma 2, Phi-3.5, and Mistral.
🚀 Asynchronous Isolate Inference: Heavy tensor calculations run on background isolates, keeping the Flutter UI running at a silky-smooth 60/120 FPS.

📱 Tested Mobile-Optimized Models

All models use the industry-standard GGUF format with Q4_K_M quantization for optimal balance between memory footprint, battery life, and accuracy:

Model	Parameter Count	File Size (Q4_K_M)	Recommended Device Profile	Best For
SmolLM2 135M	135M	~85 MB	Ultra low-end devices / budget phones	Quick intent classification, lightweight bots
SmolLM2 360M	360M	~230 MB	Any modern smartphone (2GB+ RAM)	Recommended for general mobile apps
Qwen 2.5 0.5B	490M	~380 MB	Standard smartphones	Multilingual, structured tasks, logic
Llama 3.2 1B	1.2B	~770 MB	Mid-range smartphones (4GB+ RAM)	General conversation, drafting, Q&A
SmolLM2 1.7B	1.7B	~1.0 GB	Modern flagships & tablets (6GB+ RAM)	Complex reasoning, code generation
Phi-3.5 Mini	3.8B	~2.2 GB	Flagships & Desktop (8GB+ RAM)	Deep reasoning and mathematical logic

📦 Installation

Add offline_slm to your pubspec.yaml:

dependencies:
  offline_slm: ^0.0.2

Or run:

flutter pub add offline_slm

🚀 Quickstart: Drop-in Chat UI

Drop a complete, interactive, offline AI chatbot into your app with just a few lines of code:

import 'package:flutter/material.dart';
import 'package:offline_slm/offline_slm.dart';

class OfflineChatPage extends StatefulWidget {
  const OfflineChatPage({super.key});

  @override
  State<OfflineChatPage> createState() => _OfflineChatPageState();
}

class _OfflineChatPageState extends State<OfflineChatPage> {
  late final OfflineChatController _controller;

  @override
  void initState() {
    super.initState();
    // 1. Initialize controller
    _controller = OfflineChatController(
      systemPrompt: 'You are an offline AI assistant running locally on this device.',
    );

    // 2. Load model (or prepare from preset)
    _loadModel();
  }

  Future<void> _loadModel() async {
    // Downloads and prepares SmolLM2 360M (~230MB) with caching
    final config = await ModelManager.prepareModelConfig(
      ModelPresets.smolLM2_360M,
    );
    await _controller.initializeModel(config, modelName: 'SmolLM2 360M');
  }

  @override
  void dispose() {
    _controller.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text('Offline AI Chatbot')),
      // 3. Drop in the complete chat widget
      body: OfflineChatWidget(
        controller: _controller,
        theme: OfflineChatTheme.adaptive(context),
        suggestions: const [
          'Summarize notes',
          'Help me draft an email',
          'Explain quantum computing simply',
        ],
      ),
    );
  }
}

🛠️ Headless Usage (Custom UI)

If your app already has an existing chat interface, use OfflineChatController or SLMEngine directly:

// 1. Initialize engine
final engine = LlamaEngine();
await engine.initialize(
  SLMModelConfig.auto(
    modelPath: '/path/to/local/model.gguf',
    contextSize: 2048,
  ),
);

// 2. Stream generation token-by-token
final tokenStream = engine.generateStream(
  'Write a short haiku about privacy',
  config: const GenerationConfig(
    temperature: 0.7,
    maxTokens: 256,
  ),
  onMetrics: (metrics) {
    print('Speed: ${metrics.tokensPerSecond} tokens/second');
  },
);

await for (final token in tokenStream) {
  stdout.write(token);
}

// 3. Clean up
await engine.dispose();

🧩 Prompt Templates & Stop Sequences

Different SLMs require specific format tokens. offline_slm handles this automatically:

// Automatically selects ChatML, Llama 3, Gemma, Mistral, or Phi template
final template = ChatTemplate.fromModelName('SmolLM2-360M-Instruct.gguf');

// Formats message history with required system and turn wrappers
final formattedPrompt = template.format(
  messages: chatMessages,
  systemPrompt: 'You are a concise offline assistant.',
);

Supported Templates:

ChatMLTemplate (<|im_start|>user...<|im_end|>) — SmolLM2, Qwen 2.5, Yi
Llama3Template (<|start_header_id|>...<|eot_id|>) — Meta Llama 3.2 / 3.1
GemmaTemplate (<start_of_turn>...<end_of_turn>) — Google Gemma 2
MistralTemplate ([INST]...[/INST]) — Mistral AI
PhiTemplate (<|user|>...<|end|>) — Microsoft Phi-3 / 3.5
CustomChatTemplate — Custom formatters

⚡ Performance Optimization for Mobile

Quantization Selection: Always prefer Q4_K_M GGUF models on mobile devices for optimal memory retention and token generation speed.
Context Window Size: Set contextSize between 1024 and 2048 tokens on phones with 3GB–4GB RAM to prevent memory warnings.
Hardware Acceleration:
- iOS / macOS: Offload layers to GPU (gpuLayers: 99) to leverage Apple Metal acceleration.
- Android: CPU thread tuning defaults to 4 threads (threads: 4) for multi-core ARM big.LITTLE architectures.
Instant Testing with Simulator: Use MockSLMEngine() during UI design or CI/CD test runs to preview streaming animations, typing cursors, and layout behaviors without downloading gigabyte-sized files.

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.