offline_upload_queue 1.3.3

Offline-first image upload queue for Flutter — persistent, retry-capable, background-aware.

Offline Upload Queue

Offline-first file upload queue for Flutter. Uploads survive app kills, flaky networks, and opportunistic OS background wakes.

Maintained by Cengizhan Kaya.

Requires Flutter ≥ 3.24 · Dart ≥ 3.12 · iOS & Android only

Persistence via Sembast, networking via Dio, background via Workmanager (Android) and BGTaskScheduler (iOS).

---

Contents

• Why this package
• Demo
• Features
• Installation
• Quick start
• Custom upload adapter
• Server-side idempotency
• Wi‑Fi only & force upload
• Auth token refresh
• Task state machine
• Sandbox & disk usage
• Background sync
• API highlights
• Metrics & notifications
• Caveats
• Security
• Resumable / chunked uploads
• Concurrent uploads
• Deduplication
• When not to use
• Links

---

Why this package

• Offline-first enqueue — accept files immediately; upload when the network allows.
• Crash resilient — tasks live in a local Sembast DB; stuck uploading rows recover after lock acquisition.
• Stable taskId (idempotency key) — UUID created at enqueue(), reused on every retry so your server can safely ignore duplicates after a “response lost” kill (see Server-side idempotency).
• Chunked / resumable uploads — optional supportsResumable + uploadChunk with persisted offset/session (see Resumable / chunked uploads).
• OS background drain — best-effort wakeups via BGTaskScheduler / Workmanager (not a foreground service).
• Pluggable transport — bring REST, S3, Firebase, GraphQL, or any custom UploadAdapter.
• Reactive UI — watchSummary(), watchTasks(), watchProgress().

Integration coverage includes chaos network scenarios (429 / timeouts), stale-lock handoff, large-file sandbox copy, and encryption smoke tests. See example/integration_test/.

---

Demo

From the example/ app on Android — enqueue photos and watch the reactive summary + per-task progress update live:

Live queue	After enqueue	Wi‑Fi only / force upload

---

Features

• Persistent queue across process death and reboot (Sembast, pure-Dart)
• Stable taskId per enqueue — forward to your API as an idempotency key
• Exponential backoff with jitter; permanent vs transient failure types
• Task priority (higher first) + FIFO sequenceNumber tie-break
• enqueue / enqueueBatch for single or multi-file selection
• wifiOnly + one-shot cellular bypass (forceUploadOnce)
• SHA-256 checksum (optional pin-at-enqueue)
• onAuthExpired hook for 401/403 recovery
• Disk usage estimate + warning callback
• Multiple isolated queues via boxName
• Optional DB encryption (encryptionKey) and per-field MetadataCodec
• Zero-copy sandbox via hardlink (ln) with streaming copy fallback
• Event-based lock takeover between isolates (no 30s poll wait)
• Adaptive polling under short OS background deadlines
• Optional onMetrics / onTaskTerminal hooks for observability
• Optional chunked / resumable uploads via supportsResumable + uploadChunk
• Optional parallel uploads via maxConcurrentUploads (default: serial)
• Optional checksum-based dedup via deduplicateByChecksum (default: off)

---

Installation

dependencies:
  offline_upload_queue: ^1.3.3

flutter pub get

Minimum: Flutter 3.24+, Dart 3.12+. Platforms: iOS and Android only.

---

Quick start

import 'package:offline_upload_queue/offline_upload_queue.dart';

final queue = UploadQueue(
  adapter: RestUploadAdapter(
    baseUrl: 'https://api.example.com',
    authHeaderProvider: () async => 'Bearer ${await tokenStore.getToken()}',
  ),
);
await queue.init();

final taskId = await queue.enqueue(
  filePath: photo.path,
  metadata: {'albumId': '42', 'userId': 'u1'},
);

queue.watchSummary().listen((s) {
  debugPrint('${s.pending} pending · ${s.completed} completed');
});

// App shutdown / isolate teardown:
await queue.dispose();

Full demo: example/.

---

Custom upload adapter

RestUploadAdapter posts multipart to $baseUrl/upload and includes the stable taskId in the form body (default field name taskId). For S3, Firebase, GraphQL, or a custom protocol, extend UploadAdapter (so default uploadChunk / supportsResumable are inherited) and forward taskId to your backend as an idempotency key:

class S3UploadAdapter extends UploadAdapter {
  S3UploadAdapter(this._client);
  final YourS3Client _client;

  @override
  Future<UploadResult> uploadFile({
    required String taskId, // stable across retries — use as Idempotency-Key
    required String filePath,
    required Map<String, dynamic> metadata,
    required String checksum,
    void Function(int sent, int total)? onProgress,
    UploadCancelToken? cancelToken,
  }) async {
    try {
      cancelToken?.registerOnCancel(() => _client.abort());
      final remote = await _client.putObject(
        filePath: filePath,
        key: taskId,
        onProgress: onProgress,
      );
      return UploadResult.success(remoteChecksum: remote.etag);
    } on AuthException {
      return const UploadResult.failure(FailureType.authExpired);
    } on RateLimitException catch (e) {
      return UploadResult.failure(
        FailureType.rateLimited,
        retryAfter: e.retryAfter,
      );
    } catch (_) {
      // Prefer returning failure over throwing — cancel paths also return failure.
      if (cancelToken?.isCancelled ?? false) {
        return const UploadResult.failure(FailureType.unknown);
      }
      return const UploadResult.failure(FailureType.network);
    }
  }
}

final queue = UploadQueue(adapter: S3UploadAdapter(s3));

---

Server-side idempotency

Every upload attempt includes a taskId: a UUID v4 generated once at enqueue() and kept unchanged across retries, process death, and background wakeups. RestUploadAdapter sends it as the multipart field taskId (override via onBuildFormData if you prefer a header such as Idempotency-Key).

This protects the classic offline-first failure:

1. Device sends the file; the server stores it and returns 200 OK.
2. The response is lost (app kill, tunnel drop, OS reclaim).
3. After restart the queue still sees the task as incomplete and retries with the same taskId.

Without server-side dedupe that retry creates a duplicate object. With it, the second request is a no-op that returns the original result.

What your server should do

1. Treat taskId as a unique constraint on the upload record (or as an Idempotency-Key cache entry with the first successful response body).
2. On a repeated taskId after success: return the original success response (same status / payload / remote id) — do not process the file again.
3. Optionally also store the client checksum (SHA-256) for integrity checks; that complements but does not replace taskId idempotency (checksum dedup is content-based; taskId is per enqueue).

Minimal sketch (pseudo-SQL / handler):

UNIQUE INDEX uploads(task_id);

on POST /upload:
  if exists row where task_id = :taskId and status = 'stored':
    return 200 + original remote URL   # replay
  store file, insert row(task_id, checksum, ...)
  return 200 + new remote URL

Client-side content dedup (deduplicateByChecksum) is separate — it avoids enqueueing the same bytes twice. Server idempotency on taskId avoids storing the same enqueue twice after a lost response.

---

Wi‑Fi only & force upload

final queue = UploadQueue(
  adapter: adapter,
  wifiOnly: true,
);
await queue.init();

// Later, on cellular, process the *current* pending snapshot once:
await queue.forceUploadOnce();
// Tasks enqueued after this call still wait for Wi‑Fi.

pause() takes precedence over forceUploadOnce().

---

Auth token refresh

final queue = UploadQueue(
  adapter: RestUploadAdapter(
    baseUrl: apiBase,
    authHeaderProvider: () async => 'Bearer ${await tokens.read()}',
  ),
  onAuthExpired: () async {
    await tokens.refresh(); // must complete before the next attempt
  },
  authTimeout: const Duration(seconds: 30),
);

While onAuthExpired runs, pausedDueToAuth: true is set on QueueSummary and _fillSlots will not start new tasks. In-flight uploads (when maxConcurrentUploads > 1) may still finish. Fail or timeout → normal backoff.

---

Task state machine

By default only one file uploads at a time (serial worker); set maxConcurrentUploads for parallel uploads (see Concurrent uploads).

State	Meaning
pending	Waiting for the worker
uploading	Active HTTP/upload work
completed	Success; sandbox copy deleted
failed	Transient error — will retry with backoff
permanentlyFailed	Fatal / max attempts — no auto-retry
cancelled	User cancelled — no auto-retry

Re-queue terminal tasks with queue.retry(taskId) (permanentlyFailed or cancelled only).

---

Sandbox & disk usage

Default: copyToSandbox: true.

1. Try a hardlink (ln) on the same volume (no extra disk blocks; non-Windows).
2. On failure → byte copy (File.copy or streaming above sandboxCopyThresholdBytes).

estimatedDiskUsageBytes sums logical sizes (conservative when hardlinks succeed).

UploadQueue(
  adapter: adapter,
  advanced: UploadQueueAdvancedOptions(
    diskUsageWarningBytes: 100 * 1024 * 1024,
    onDiskUsageWarning: (current, limit) {
      // Prompt the user to purge completed / cancelled tasks
    },
  ),
);

---

Background sync

Background work is best-effort. iOS schedules opportunistically; Android is subject to Doze. This package does not run a foreground service.

Shared vs owned queue

Context	Pattern
Android Workmanager isolate	Create a new UploadQueue → BackgroundTaskRunner.run disposes it
iOS BGTask on the app engine	Pass the same foreground queue → runner does not dispose it
iOS expiration	Call queue.abortActiveUploads() — never dispose() the shared queue

iOS (BGTaskScheduler)

1. Info.plist

<key>BGTaskSchedulerPermittedIdentifiers</key>
<array>
  <string>com.example.app.upload_refresh</string>
  <string>com.example.app.upload_processing</string>
</array>

2. Xcode → Signing & Capabilities → Background Modes

• Background fetch
• Background processing

3. Register in AppDelegate.swift (see example/ios/Runner/AppDelegate.swift)

4. Dart

IosBackgroundChannel.instance.setMethodCallHandler(
  onAppRefresh: () => BackgroundTaskRunner.run(queue),
  onProcessing: () => BackgroundTaskRunner.run(queue),
  onExpiration: () {
    queue.abortActiveUploads(); // keep the foreground queue alive
  },
);

Android (Workmanager)

1. AndroidManifest.xml

<uses-permission android:name="android.permission.RECEIVE_BOOT_COMPLETED"/>
<!-- No FOREGROUND_SERVICE permission required -->

2. Top-level dispatcher

@pragma('vm:entry-point')
void callbackDispatcher() {
  Workmanager().executeTask((taskName, inputData) async {
    WidgetsFlutterBinding.ensureInitialized();
    if (taskName == AndroidBackgroundRunner.taskName) {
      final queue = UploadQueue(adapter: MyAdapter());
      final hasPending = await BackgroundTaskRunner.run(queue);
      if (hasPending) await AndroidBackgroundRunner.scheduleNextRun();
      return true;
    }
    return false;
  });
}

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  Workmanager().initialize(callbackDispatcher);
  runApp(const MyApp());
}

After enqueueing work in the foreground, call AndroidBackgroundRunner.scheduleNextRun() when you want a background drain chain.

---

API highlights

API	Role
init / dispose	Open DB + worker; safe to init again after dispose
enqueue / enqueueBatch	Add files (+ JSON metadata, optional priority)
pause / resume	In-memory worker gate
forceUploadOnce	Cellular bypass snapshot
cancel / retry / getTask / findByChecksum	Per-task control + checksum lookup
purge / purgeAll*	Delete terminal (or completed) rows + sandbox files
abortActiveUploads	Cancel in-flight work → pending without disposing
watchSummary / watchTasks / watchProgress	Reactive UI

Use list index for UI order — sequenceNumber may have gaps.

API docs: pub.dev documentation.

---

Metrics & notifications

Counters live for the process lifetime (not persisted). Snapshots fire on each heartbeat and when uploads complete or permanently fail:

UploadQueue(
  adapter: adapter,
  advanced: UploadQueueAdvancedOptions(
    onMetrics: (m) {
      // Optional: Sentry / Prometheus
      // Sentry.configureScope((s) => s.setTag('uploads_ok', '${m.uploadsSucceeded}'));
      debugPrint('queue metrics: $m');
    },
    onTaskTerminal: (task) {
      // Wire flutter_local_notifications yourself — the package stays dependency-free.
      if (task.status == UploadStatus.completed) {
        // showNotification('Upload done', task.taskId);
      }
    },
  ),
);

---

Caveats

1. copyToSandbox: false — you must keep the source file alive. Do not pass raw content:// / PHAsset URIs; resolve to a real filesystem path first.
2. Checksum timing — by default checksum runs at upload start. Set pinChecksumAtEnqueue: true to pin earlier (slower enqueue on large files).
3. staleLockThreshold — default 5 minutes; must be ≥ heartbeatInterval * 3. Increase if uploads routinely exceed the threshold, or another worker may steal the lock.
4. Serial by default — one upload at a time unless maxConcurrentUploads > 1 (see Concurrent uploads).
5. enqueueBatch is not atomic — items are added in order and the first invalid entry (missing file, non-JSON metadata) throws ArgumentError; earlier items stay queued, later ones are not added. Validate paths up front if you need all-or-nothing.
6. cancel is a no-op on terminal tasks — already completed / permanentlyFailed / cancelled tasks are left untouched, and onTaskTerminal fires only once per task.
7. deduplicateByChecksum hashes eagerly — like pinChecksumAtEnqueue, it reads the whole file at enqueue() time; matches completed (new taskId) or active pending/uploading/failed (alias); completed-hit memory is forgotten after purgeAllCompleted() (see Deduplication).

---

Security

• encryptionKey — encrypts the Sembast file with an unaudited Salsa20+SHA256 sample codec (no MAC/AEAD). Prefer OS disk encryption and avoid storing PII in metadata for strict compliance.
• Audited alternative — for GDPR/KVKK/HIPAA-style requirements, keep PII out of the queue DB, rely on OS file-based encryption (iOS Data Protection / Android credential-encrypted storage), and/or supply your own PersistenceRepository backed by an independently audited store. The bundled codec is a convenience sample, not a compliance control.
• MetadataCodec — encrypt only the metadata field without encrypting the whole DB.
• Reachability — default probe is https://connectivitycheck.gstatic.com/generate_204. Override with DefaultConnectivityMonitor(reachabilityUrl: '...').

---

Resumable / chunked uploads

Supported since v0.8.0 (opt-in via your adapter). This is not whole-file-only: large files can resume after process death.

If your UploadAdapter sets supportsResumable => true and overrides uploadChunk, files ≥ chunkThresholdBytes (default 20 MiB) are split into chunkSizeBytes pieces (default 8 MiB). After each accepted chunk the queue persists:

• UploadTask.bytesUploaded — absolute byte offset accepted by the server
• UploadTask.resumableSessionId — opaque remote session (tus URL, S3 uploadId, etc.)

On the next attempt (retry, crash recovery, or new process) the worker calls uploadChunk again with that offset + sessionId instead of restarting from byte 0. Smaller files still use uploadFile (whole-file path). RestUploadAdapter keeps supportsResumable == false by default.

class MyTusAdapter extends UploadAdapter {
  @override
  bool get supportsResumable => true;

  @override
  Future<UploadChunkResult> uploadChunk({
    required String taskId,
    required String filePath,
    required Map<String, dynamic> metadata,
    required String checksum,
    required int offset,
    required int chunkSize,
    String? sessionId,
    void Function(int sent, int total)? onProgress,
    UploadCancelToken? cancelToken,
  }) async {
    // PATCH/PUT the [offset, offset+chunkSize) range; return bytesAccepted
    // and sessionId so the queue can resume after a kill.
    ...
  }
}

UploadQueue(
  adapter: MyTusAdapter(...),
  advanced: UploadQueueAdvancedOptions(
    chunkThresholdBytes: 20 * 1024 * 1024,
    chunkSizeBytes: 8 * 1024 * 1024,
  ),
);

Picking a chunk size: S3 multipart requires ≥ 5 MiB per part (last part exempt, max 10,000 parts); 5–16 MiB is a reasonable range on mobile networks. Values below 1 throw ArgumentError at init().

An adapter that returns success with bytesAccepted: 0 and complete: false makes no progress; the queue treats that as a transient failure and applies normal backoff instead of looping forever.

Reference adapters (not production-hardened): example/lib/adapters/tus_upload_adapter.dart, example/lib/adapters/s3_multipart_upload_adapter.dart.

Unit coverage: test/resumable_upload_test.dart.

Cancel note: local offset/session are cleared on cancel / permanent corruptFile; the remote tus/S3 session may remain — clean it up in your adapter or backend if needed. For whole-file adapters, rely on server-side idempotency so a full retry after a lost 200 OK does not create a second object.

---

Concurrent uploads

By default the worker is serial — one upload at a time. Set maxConcurrentUploads to process several files in parallel:

UploadQueue(
  adapter: adapter,
  advanced: UploadQueueAdvancedOptions(
    maxConcurrentUploads: 3,
  ),
);

The queue still orders by priority DESC, sequenceNumber ASC, but the guarantee becomes "top-N start together" rather than strict one-at-a-time completion: with maxConcurrentUploads: 2, the two highest-priority eligible tasks start immediately; as soon as either slot frees up, the next eligible task takes it. cancel() / retry() / getTask() are unaffected — they already operate per-task regardless of how many uploads are active.

maxConcurrentUploads: 1 (default) reproduces the original serial behavior exactly. Values below 1 throw ArgumentError at init().

---

Deduplication

Enable deduplicateByChecksum to skip re-uploading (or double-queueing) a file whose content (SHA-256 checksum) already matches a task in the DB:

UploadQueue(
  adapter: adapter,
  advanced: UploadQueueAdvancedOptions(
    deduplicateByChecksum: true,
  ),
);

Behavior when enqueue() finds a match:

Existing status	enqueue() returns	Effect
completed	new unique taskId	Task starts completed immediately (no network); onTaskTerminal fires
pending / uploading / failed	same taskId (alias)	No new DB row / sandbox copy; watch the existing task
cancelled / permanentlyFailed	new pending task	No match — upload proceeds normally

UploadQueueMetrics.uploadsDeduplicated increments on both completed-hit and active-alias. Query the store yourself with queue.findByChecksum(checksum, statuses: {...}).

Notes:

• Matching is content-only (checksum); metadata and destination are ignored. On a completed-hit the new task keeps its own metadata.
• Completed-hit "memory" lasts only while the matching completed row stays in the DB — purgeAllCompleted() / purge() erase it.
• Cost: every enqueue() hashes the full file up front (same cost class as pinChecksumAtEnqueue).
• Default: false (fully backward compatible when left off).

---

When not to use

• You need web / desktop — this package targets iOS & Android only.
• A single fire-and-forget request with no offline queue is enough — use Dio directly.
• You require a guaranteed background upload SLA — no mobile OS provides that without user-visible foreground work.

---

Links

• pub.dev
• API reference
• Example app
• Changelog
• Migration
• Integration test matrix
• Contributing
• Issues
• Author: Cengizhan Kaya

License

MIT © Cengizhan Kaya — see LICENSE.