masamune_experiment 3.0.1

Plug-in for Masamune that defines A/B tests and feature switches in a single synchronous function based on Firebase Remote Config values.

Masamune Experiment

---

[GitHub] | [YouTube] | [Packages] | [X] | [LinkedIn] | [mathru.net]

Masamune Experiment

Define A/B tests and feature switches in a single synchronous function based on Firebase Remote Config values.

• Values are loaded through masamune_model_firebase_remote_config and kept in memory, so branches are evaluated synchronously.
• Every branch has a label, and the first branch whose condition matches is executed.
• Branches can be fixed with overrides for debugging and tests, or with fixed after an A/B test is finished.
• A/B tests are performed only on the client. See Server Side.

Usage

Installation

1. Add the package to your project.

flutter pub add masamune_experiment

Or enable firebase.experiment.enable in katana.yaml and run katana apply.

firebase:
  experiment:
    enable: true

Setup

2. Register ExperimentMasamuneAdapter. Pass FirebaseOptions or a FirebaseRemoteConfig instance to FirebaseRemoteConfigModelAdapter. initialValue is used before loading and when values cannot be fetched, so set the control values there.

// lib/main.dart

import 'package:masamune_experiment/masamune_experiment.dart';

final masamuneAdapters = <MasamuneAdapter>[
  const UniversalMasamuneAdapter(),

  ExperimentMasamuneAdapter(
    modelAdapter: FirebaseRemoteConfigModelAdapter(
      options: DefaultFirebaseOptions.currentPlatform,
      initialValue: {
        "new_button": "control",
      },
    ),
  ),
];

Load Values

3. Get the Experiment controller and call load() at the beginning of the page. When loading is complete, the page is rebuilt.

class HomePage extends PageScopedWidget {
  @override
  Widget build(BuildContext context, PageRef ref) {
    final experiment = ref.app.controller(Experiment.query())..load();

    // ...
  }
}

Define Branches

4. Define all branches inside Experiment.run.

• The first argument receives ExperimentConfig (same getters as FirebaseRemoteConfig) and returns the value used for the conditions. It is evaluated only once.
• The second argument is the list of ExperimentCase. Each case has a label, a condition when, and a process run. Cases are evaluated from the top, and only the first matching run is executed.
• The return value of run becomes the return value of Experiment.run, so you can return widgets or data. If run returns a Future, Experiment.run returns the Future.
• fixed executes the case with the same label regardless of the conditions. Use it to fix the winner after an A/B test.

final button = experiment.run(
  (config) => config.getString("new_button"),
  [
    ExperimentCase.equals(
      "treatment",
      label: "treatment",
      run: () => const NewButton(),
    ),
    ExperimentCase(
      label: "treatment_large",
      when: (value) => value.startsWith("large"),
      run: () => const LargeButton(),
    ),
    ExperimentCase.otherwise(
      label: "control",
      run: () => const OldButton(),
    ),
  ],
  // fixed: "treatment",
);

Experiment.run is synchronous. Load values in advance with Experiment.load. Before loading, initialValue is used.

If the return types of the cases differ, specify type arguments explicitly, such as run<String, Widget>(...).

Fix Branches for Debugging and Tests

overrides always take precedence over the loaded values. Pass them only in debug or test flavors.

ExperimentMasamuneAdapter(
  modelAdapter: FirebaseRemoteConfigModelAdapter(
    options: DefaultFirebaseOptions.currentPlatform,
    initialValue: {"new_button": "control"},
  ),
  overrides: {
    if (const String.fromEnvironment("EXPERIMENT_NEW_BUTTON").isNotEmpty)
      "new_button": const String.fromEnvironment("EXPERIMENT_NEW_BUTTON"),
  },
);

Testing

Write at least one test for every ExperimentCase label. ExperimentMasamuneAdapter.test does not use Firebase, and Experiment.selectedLabels returns the labels selected by run.

for (final label in ["treatment", "control"]) {
  test("selects $label", () {
    final experiment = Experiment(
      adapter: ExperimentMasamuneAdapter.test(
        overrides: {"new_button": label},
      ),
    );
    buildButton(experiment);
    expect(experiment.selectedLabels, [label]);
  });
}

For widget tests and golden tests, pass ExperimentMasamuneAdapter.test(overrides: {...}) to the adapters of the app under test and capture each branch.

Server Side

A/B tests are not performed on the server. The client and the server may evaluate conditions differently, so branching on both sides is risky.

When a branch on the client needs server changes, the server must support the requests of all branches (backward compatible) and test all of them.

GitHub Sponsors

Sponsors are always welcome. Thank you for your support!

https://github.com/sponsors/mathrunet