daybook_health

Errors and app health for a Flutter app, sent to Daybook's Health hub. The Dart errors your app does not catch are sent by themselves, with their stack as readable frames and your own code marked, the device, the OS, the app's version and build, the signed-in user, your tags and the breadcrumbs that led there. Errors you catch go with one call. Each run of the app is a session, so the hub can say how many sessions and users went without a crash in each release. On iOS, native crashes (a Swift trap, a bad access, an uncaught NSException) are recorded too and sent on the next launch, the main thread is watched for hangs, and a launch finds out whether the system ended the last run for memory. On Android, uncaught Java and Kotlin exceptions are recorded as the app dies, and from Android 11 the ANRs, native crashes and memory kills the system recorded are sent on the next launch.

Written in Dart, with a small native part on iOS and Android that says what the device and the app are, on iOS records native crashes with KSCrash (MIT, its Recording part, pinned at 2.6.1, pulled in by CocoaPods or Swift Package Manager with the plugin), and on Android records JVM crashes and reads the system's exit records with no library at all. No other packages. iOS 15, Android API 24.

Get a key

In Daybook, open the project's settings, find the app under App key and issue its key: one for each build kind (Production, Staging, Development). The key starts with dbk_ and is public by design: it can only send, and it says which app, of which project, in which environment. Every Daybook module in the app uses the same one. The environment comes from the key; nothing in the app sets it.

Install

Health is a module of Daybook. Add the core and the module:

flutter pub add daybook daybook_health
dependencies:
  daybook: ^0.1.0
  daybook_health: ^0.1.0

The iOS app must target iOS 15 or later: flutter create starts at 13, so raise IPHONEOS_DEPLOYMENT_TARGET in ios/Runner.xcodeproj (and platform :ios in a Podfile, if there is one).

Start

import 'package:daybook/daybook.dart';
import 'package:daybook_health/daybook_health.dart';

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();
  final daybook = await Daybook.initialize('dbk_…');
  runApp(const App());
}

Once, at launch, as early as you can: errors before it are not caught. Health starts with Daybook; the app names no module. Later, anywhere: Daybook.instance.health.

From then on Daybook catches, and sends as unhandled:

  • what FlutterError.onError gets: errors in building, layout, painting and the framework's callbacks;
  • what PlatformDispatcher.instance.onError gets: errors nothing else caught, asynchronous ones above all;
  • what reaches the root isolate's error listener.

A handler your app set before Daybook.initialize still runs, after Daybook's, and Flutter still prints the error to the console as it did. If your app runs inside a zone of its own, pass Daybook's handler to it:

runZonedGuarded(() => runApp(const App()), DaybookHealth.captureUnhandled);

Send a test error

Daybook.instance.health.sendTestError();

It sends a handled error, DaybookHealthTestError, that says it is a test, tagged daybook.test: true. It shows in the Health hub under the app's source within a few seconds; if it does not, run with Daybook.initialize(key, logging: true) and read what the console says.

Errors you catch

try {
  await checkout(cart);
} catch (error, stackTrace) {
  Daybook.instance.health.captureException(error, stackTrace, tags: {'step': 'payment'});
}
captureException
error, stackTrace What catch (error, stackTrace) gives. Without a stack the error goes without frames.
handled true by default: the app caught it and went on.
tags, contexts Added to the ones set with setTag and setContext, for this error.
fingerprint Your own grouping for this error, instead of Daybook's.
fatal The process is ending because of it: the error is fatal and the session crashed.

It answers the event's id, or null when nothing was sent. captureMessage('…', level: HealthLevel.warning) sends a line with no exception.

final health = Daybook.instance.health;
health.addBreadcrumb('Tapped pay', category: 'ui', data: {'items': 3});
health.addHttpBreadcrumb(method: 'GET', url: uri, statusCode: 200, duration: elapsed);
health.setTag('plan', 'pro');
health.setContext('cart', {'items': 3, 'currency': 'EUR'});

The last 100 breadcrumbs go with each error. The screens the person went through are breadcrumbs too when the navigator has Daybook's observer, which also says which route the error happened on:

MaterialApp(navigatorObservers: [HealthNavigatorObserver()], …)

Route names only, never their arguments. An HTTP breadcrumb drops the address's query and fragment, where tokens travel; call addHttpBreadcrumb from your HTTP client's interceptor.

Options

Set before Daybook starts:

DaybookHealth.configure(
  inAppPackages: {'shop', 'shop_core'},
  beforeSend: (event) => event.tags['internal'] == 'true' ? null : event,
);
configure parameter Default
enabled true false: the module does nothing at all
sampleRate 1.0 The share of errors sent; sessions are always sent and still count every error
inAppPackages found The Dart packages that are your app's own code (below)
beforeSend none Change an error before it is sent, or answer null to drop it. If it throws, the error is dropped
maxBreadcrumbs 100 How many breadcrumbs are kept, at most 100
captureUnhandledErrors true Install the three handlers above
reportSilentFlutterErrors false Flutter's silent errors (an image that did not load in a release build)
sessions true A session for each run of the app
captureNativeCrashes true Record native crashes, hangs and memory kills (iOS), JVM crashes, ANRs, native crashes and memory kills (Android), and send them (below). false: none of it
captureHangs true iOS: watch the main thread for hangs (below). Needs captureNativeCrashes
hangThreshold 2 s iOS: how long the main thread may not answer before it is a hang; at least 100 ms
captureMemoryKills true iOS: find out on the next launch whether the system ended the last run for memory (below). Needs captureNativeCrashes

Your own code. The Health hub groups errors by the frames of your app's own code, so they are marked in_app. Daybook finds your package from the stack your main started it on, and from the app's id (com.example.shop is shop, as flutter create names both). Frames of Dart, Flutter, Daybook and other packages are not yours. When the app's code is in more than one package, or the guess is wrong, say so with inAppPackages.

Sessions

A session is one run of the app as the person sees it. It starts at launch, and when the app comes back after more than 30 seconds in the background; it ends exited after 30 seconds in the background, or when the app is closed. Its errors are counted.

A Dart error does not end a Flutter app, so it is not a crash: the session is crashed only when the app says the process is ending (captureException(…, fatal: true)), or when the plugin recorded a native crash (iOS) or a JVM or native crash (Android) in it. A session an ANR (Android 11 and later) or a watchdog termination (iOS: the app was ended while its main thread was stuck) ended is sent as hang, and one a memory kill ended as memory: neither is ever counted as a crash. A session the next launch finds still open with none of these was ended by something the app did not see and is sent as abnormal. A crash, a hang or a memory kill is not one of the session's counted errors, and a hang the app came back from ends nothing.

Native crashes on iOS

The plugin installs KSCrash's recorder when Daybook starts, with consent: Mach exceptions (a bad access, a Swift trap such as a force unwrap of nil or an index out of range), signals, uncaught NSExceptions and C++ exceptions, in the app's native code, a plugin's or the engine's. As the app dies KSCrash writes a report. On the next launch the plugin makes it into a crash event and hands it to the Dart side, which queues it with the errors and sends it through the same door, with the same key and install id, through consent, sampleRate and beforeSend. The event carries:

  • the exception's name and reason, or the Mach exception and the signal, with the Swift runtime's message when the build kept one;
  • the crashed thread's frames as addresses in images, the app's own (inside its bundle, but not the engine's Flutter.framework, KSCrash's or the plugin's own framework) marked in_app, system frames named on the device;
  • the loaded images with their debug_id (the LC_UUID), arch, load address and size, those the frames are in first; those no frame is in are left out first when the event is over the 64 KB an event may take on disk;
  • the release, device, OS, user, session, tags and last 50 breadcrumbs of the run that crashed, not of the launch that sends it.

The report is deleted once the Dart side has the event on disk; its id is the event's, so a crash handed over twice is queued once. Its frames are named once the build's dSYM is uploaded to Daybook (see Symbol files). A debugger attached to the app catches crashes before any recorder: run without Xcode's debugger (flutter run --release, or start the app from the home screen) to see one reported. An app with another crash reporter should set captureNativeCrashes: false.

Not yet on iOS: crashes before Daybook.initialize has run. Verified on an iPhone simulator only, not on a device, in an App Store build or on arm64e.

Hangs and memory kills on iOS

Hangs. With consent, a thread of the plugin's own sends the main thread (Flutter's platform thread) a ping a few times a second and waits for the answer. When none comes for hangThreshold (2 seconds unless you set it), the main thread is stuck: the plugin takes every thread's stack there and then, the main thread's whole and up to 12 frames of each of up to 24 others, with KSCrash's backtrace (each thread is stopped only while its stack is read). One hang, however long, is one event:

  • when the main thread answers again, a hang event at level error is handed to the Dart side at once and sent like any error, with how long it lasted;
  • when the app is ended while it is stuck (iOS's watchdog ends an app that does not answer for a while, above all at launch and when it goes to the background or comes back), the next launch sends the same hang at level fatal, its duration up to when the app was last seen, and its session ends as hang.

The event's frames are the main thread's, as addresses in images with their debug_id, read with the dSYM like a crash's; the other threads are in contexts.threads (the console shows them under the stuck one); contexts.hang says the threshold, the duration and whether the app was ended in it. It carries the release, user, session, tags and breadcrumbs of the run. The exception is always App Hang, so hangs at the same place are one issue. The watchdog rests in the background, does not count time the app was suspended, and does not run while a debugger is attached (a breakpoint stops the main thread).

Memory kills. iOS tells an app nothing before it ends it for using too much memory, so the plugin reads it the way Sentry does, by ruling out everything else. It keeps a small file of the run's state (Application Support/daybook_health/run/run.json): whether the app is active on screen, whether it was told it is closing, a hang going on, the app's version and build, the OS's build, when the device started, whether a debugger is attached, the memory the app holds (read every 5 seconds), the memory warnings, and the run's scope and breadcrumbs. On the next launch the last run was a memory kill when it was active on screen, did not crash, was not hung, was not closed, no debugger was attached, and neither the app's version, the OS nor the device's start time changed. It is then sent as a memory event at level fatal, with no stack, with what the app last knew of its memory in contexts.memory (footprint, its peak, the limit iOS gave, the device's memory, the warnings and when the last came) and the run's breadcrumbs, and its session ends as memory.

What this cannot tell apart, so read memory kills as "ended by the system, most likely for memory":

  • the app calling exit() or abort() without a crash being recorded, or a crash KSCrash could not write;
  • the system ending the app on screen for another reason: too much CPU, a thermal emergency, a background task that ran out of time just as it came back;
  • the app ended while it was being launched and not yet active, which is not counted at all.

A run that resigned active (the app switcher, a call, Control Center) and was then ended is not counted, so closing the app from the switcher is never a memory kill. Nothing of this is kept or sent without consent; taking consent back deletes the file. MetricKit's hang and crash diagnostics are not used yet.

The Swift files that read KSCrash's reports, watch for hangs and read the last run are shared with the Swift SDK. ios/daybook_health/Sources/daybook_health/CrashReport.swift, CrashRecorder.swift, AppHangs.swift and LastRun.swift are copies of sdk/swift/DaybookAnalytics/Sources/DaybookHealth/'s in Daybook's repository, kept the same by a test (test/native_crash_test.dart): change them there and copy them here.

Crashes, hangs and memory kills on Android

Uncaught Java and Kotlin exceptions, on any thread. When Daybook starts with consent the plugin installs a default uncaught-exception handler in front of the one that was there (Android's, or another reporter's). When an exception reaches it, it writes the event to the app's own no-backup folder at once, with no network, then hands the exception on, so the app dies exactly as it would have. The event is a crash, fatal, unhandled, with:

  • the exception's class and message, and its causes (each Type: message (at the first frame of yours)) in contexts.crash.causes;
  • the crashing thread's frames: class, method, file and line, yours marked in_app (the app's id and the package of the activity it opens on; never Flutter's, Android's, Java's or Daybook's), a native method said as Native Method; the thread's name, and every thread's name, in contexts.crash;
  • the release, device, OS, app, user, session, tags and last 50 breadcrumbs the Dart side last told the plugin, kept in memory and on disk as the run goes;
  • the build's R8 mapping id as a proguard image (below), so a release build's renamed classes are read back.

What the system recorded (Android 11, API 30, and later). At each start the plugin reads the app's process exits since the last read (ActivityManager.getHistoricalProcessExitReasons):

Exit Sent as With
REASON_ANR hang the main thread's stack (Java frames, and native frames by library and build id) from the system's ANR dump, and the dump's threads and their states in contexts.hang
REASON_CRASH_NATIVE crash from Android 12, the tombstone: the signal, its code and fault address, the abort message, the crashing thread's frames as addresses in their libraries, each library an elf image with its build id (code_id) and the debug id derived from it, so the server reads them with the unstripped .so files uploaded for the build; on Android 11, the signal and what the exit said
REASON_LOW_MEMORY (and an excessive-resource kill that names memory) memory, no stack only when the person could see the app; a memory kill in the background is how Android frees memory and only ends its session as memory
REASON_CRASH crash only when the handler above did not write that run's crash (one before Daybook started)

Each is reported as the run it happened in (matched by the process id and the process's start time), with that run's release, user, session and breadcrumbs, and only when that run had consent. The session it ended is crashed, hang or memory accordingly. Exits before the first run with this version, and the person closing the app or the system closing it in the background, are not reported.

Kept and handed over as on iOS. At the next start the plugin hands every event to the Dart side, through the same channel as the iOS plugin, and Dart queues it in its outbox (consent, sampleRate and beforeSend apply, the same key and install id); once it is on disk there, the plugin's copy is deleted. An event's id is fixed (a JVM crash's is made as it is written, an exit's from its process and time), so one handed over twice is sent once. At most 10 events and 2 MB wait in the plugin's folder, the oldest going first; one that cannot be read is dropped and logged. Without consent the handler is not installed (or is taken out again, the previous one put back), nothing is recorded, and everything kept, the runs included, is deleted.

The R8 mapping id. A release build renames classes. The app names its mapping with an id it puts in its manifest, and uploads the mapping under the same id. Daybook does not ship a Gradle plugin; these few lines do it. In android/app/build.gradle.kts:

import java.util.UUID // at the top of the file

// One id per build for the R8 mapping: DAYBOOK_MAPPING_ID when the build gives one, otherwise a new UUID,
// written to build/daybook-mapping-id.txt for the upload.
val daybookMappingId: String = System.getenv("DAYBOOK_MAPPING_ID")?.takeIf { it.isNotBlank() }
    ?: UUID.randomUUID().toString()
layout.buildDirectory.file("daybook-mapping-id.txt").get().asFile.apply {
    parentFile.mkdirs()
    writeText(daybookMappingId)
}

android {
    defaultConfig {
        manifestPlaceholders["daybookMappingId"] = daybookMappingId
    }
}

and in android/app/src/main/AndroidManifest.xml, inside <application>:

<meta-data android:name="team.daybook.health.mapping_id" android:value="${daybookMappingId}" />

Then, after flutter build apk --release (or appbundle), dart run daybook_health:upload_symbols uploads the mapping under the id the build put in the app (read from the merged manifest, or build/app/daybook-mapping-id.txt), with the unstripped native libraries: see Symbol files.

(Or set DAYBOOK_MAPPING_ID yourself, android-2.4.0-118 for instance, for the build and the upload.) The id may have letters, digits, ., _ and -, at most 128; another is not sent, and the plugin says so in the log. Without the meta-data JVM events carry no proguard image and a release build's classes stay renamed.

Not captured on Android:

  • native crashes before Android 11: the plugin has no signal handler of its own (no NDK code), so a crash in C or C++ code (the engine, a plugin's library, the app's own) on Android 7 to 10 is not reported; its session is sent as abnormal;
  • ANRs and memory kills before Android 11, for the same reason;
  • on Android 11, the native crash's stack (Android gives the tombstone from Android 12);
  • a JVM crash before the Flutter engine has attached the plugin and Dart has started Daybook, beyond what the system's record says of it;
  • processes of the app other than its main one.

Verified with JVM unit tests (an ANR dump, tombstones in protobuf and text, the store and its limits) and builds of the example; not yet on a device or an emulator.

Obfuscated and split-debug-info builds

A build made with --obfuscate or --split-debug-info has no names in its stacks, only addresses. Daybook sends them as they are, with the snapshot's build id and where it was loaded, and keeps the stack's text in the error's contexts. On Android the snapshot is libapp.so, an ELF: its image is sent as elf with the build id as its code_id and the debug id derived from it, which is how the .symbols file of --split-debug-info is found. The Health hub makes them readable once the build's symbols are uploaded.

An --obfuscate build renames types too, so an exception's type reads aB rather than CartException. Every Dart event of such a build names the build's obfuscation map as a dart_map image whose id is the snapshot's build id (the build_id in the stack's header, the same as the .symbols file's ELF build id or Mach-O UUID); the uploader below uploads the map under that id, and the hub reads the type back. A build without --obfuscate sends no such image.

Symbol files

Release builds ship frames nobody can read: Dart's addresses, R8's renamed classes, native addresses on iOS and Android. The package has a command that uploads the build's symbol files to Daybook, after which the Health hub reads frames of that build as function, file and line, and the events that arrived before are made readable too. Run it in the app's folder, after the build:

flutter build ipa --obfuscate --split-debug-info=build/symbols \
    --extra-gen-snapshot-options=--save-obfuscation-map=build/obfuscation.map.json
flutter build appbundle --obfuscate --split-debug-info=build/symbols \
    --extra-gen-snapshot-options=--save-obfuscation-map=build/obfuscation.map.json

export DAYBOOK_SYMBOLS_TOKEN=dhs_…   # from the build's secrets
dart run daybook_health:upload_symbols

The token is the source's symbol upload token (Health, the source's settings, Symbol files): it can only upload symbol files. It is read from DAYBOOK_SYMBOLS_TOKEN and never from the command line, where it would end up in shell history and build logs; without it the command stops. DAYBOOK_SERVER (or DAYBOOK_URL) is Daybook's API, https://api.daybook.team when unset.

With no options it looks in the places Flutter writes to and uploads what it finds:

What Where Sent as
Dart's debug information (--split-debug-info), one file per architecture build/symbols, build/app/outputs/symbols (*.symbols) elf (Android) or macho (iOS), found by its build id
Dart's obfuscation map (--save-obfuscation-map) build/obfuscation.map.json, build/obfuscation-map.json, or a *obfuscation*.json in build/ or the symbols folder dart_map, once for each .symbols file's build id: the id the app sends
The R8 mapping build/app/outputs/mapping/release/mapping.txt proguard, with the id in the release build's merged manifest (team.daybook.health.mapping_id), DAYBOOK_MAPPING_ID or build/app/daybook-mapping-id.txt; left out, with a note, when there is none
Plugins' unstripped native libraries build/app/intermediates/merged_native_libs/release (*.so, not libflutter.so or libapp.so; a stripped one is left out) elf
dSYMs: Runner, App, Flutter, plugins build/ios/archive/*.xcarchive/dSYMs, build/ios/Release-iphoneos, build/ios/iphoneos macho, each DWARF file; App.framework.dSYM is left out when app.ios-arm64.symbols is the same object

Options:

--version V, --build B the app's version and build (default: pubspec.yaml's version: 2.4.0+118)
--source ID the source the token must be for (the server checks)
--server URL instead of DAYBOOK_SERVER
--split-debug-info DIR, --obfuscation-map FILE, --mapping FILE, --native-libs DIR, --dsym PATH where that kind of file is, instead of the standard places (the repeatable ones may be given more than once)
--mapping-id ID the R8 mapping's id
--build-id ID name the obfuscation map by this id rather than by the .symbols files' build ids
--dry-run list each file with its kind, id and size; send nothing (no token needed)

Files and folders named after the options are added (.symbols, .so, .dSYM). Each file is sent in three calls (a ticket, the bytes straight to storage, "done"); a network error or a 5xx is tried again three times. Each line says what became of a file: sent, replaced (the hub had that build's file already) or refused, with the hub's reason. The exit status is 0 when every file was recorded, 1 when one was refused or could not be sent, 2 for a wrong command line or no token.

On CI, the same two lines after the build steps, with the token in the CI's secrets. Run it once per build that is shipped: a later build overwrites build/.

  • Errors and session updates wait on disk and are sent within a couple of seconds, every minute, when the app goes to the background, at the next launch, and on flush().
  • A batch that could not be sent is sent again exactly as it was, after 1 s, 2 s, 4 s … up to five minutes, or after the server's Retry-After. Each error has an id made on the device, so a retry is stored once.
  • At most 30 errors and 200 KB wait on disk; past that the oldest go first, so an app crashing in a loop with no network cannot fill the phone. An error is held under the server's limits before it is kept.
  • Health measures, so it follows the person's tracking consent, the core's: Daybook.initialize(key, trackingConsent: false) and daybook.setTrackingConsent(true). Without it nothing is captured, kept or sent, and what was waiting is deleted when consent is taken back.
  • A key the server refuses stops sending until the next launch, and says so in the console.

Privacy

  • No IP address is sent or read, no device name, no advertising id and no identifier for vendor.
  • With each error: the app's id, version and build, the device model and family, whether it is a simulator, the OS and its version, the user id only when the app identified the user through daybook.identify, the install id (in a header), and what the app added itself.
  • The server removes secrets it recognizes from every text the app wrote. Use beforeSend for anything else that must not leave the phone.
  • The iOS privacy manifest declares crash and diagnostic data for app functionality, not linked to the user, and the user id, linked to the user; no tracking and no tracking domains; the file timestamp API (reason C617.1), which KSCrash's recorder uses on its own files, as KSCrash's own manifests declare; and the system boot time API (reason 35F9.1): the hang watchdog measures time with the system's uptime, and the run's state compares the device's start time between two runs. Neither is sent.

Platforms

iOS 15 and later, Android 7 (API 24) and later; ANRs, native crashes and memory kills from Android 11 (API 30), a native crash's stack from Android 12 (API 31). Declared permission: INTERNET.

Libraries

daybook_health
Errors and app health for Flutter on Daybook (PM-R-0114), as a module of Daybook (PM-R-0110): Daybook.initialize starts it, and daybook.health is its door.