Client Implementation Guidelines for Experiments

Note: This guidance is useful for implementing Normandy experiments. To ship experiments with the Nimbus platform, please see the guidance for engineers at https://experimenter.info/.

There are three supported approaches for enabling experimental features for Firefox:

Firefox Prefs
- Prefs can be used to control features that land in-tree. Feature Gates provide a wrapper around prefs that can be used from JavaScript.
Firefox Extensions AKA "Add-ons".
- If the feature being tested should not land in the tree, or if it will ultimately ship as an extension, then an extension should be used.

New features go through the standard Firefox review, testing, and deployment processes, and are then enabled experimentally in the field using Normandy.

Prefs

Firefox Preferences (AKA "prefs") are commonly used to enable and disable features. However, prefs are more complex to implement correctly than feature gates.

Each pref should represent a different experimental treatment. If your experimental feature requires multiple prefs, then Normandy does not currently support this but will soon. In the meantime, an extension such as multipreffer may be used.

There are three types of Prefs:

Built-in prefs - shipped with Firefox, in firefox.js.
user branch - set by the user, overriding built-in prefs.
default branch - Overrides both built-in and user branch prefs. Only persists until the browser session ends, next restart will revert to either built-in or user branch (if set).

Normandy supports overriding both the user and default branches, although the latter is preferred as it does not permanently override user settings. default branch prefs are simple to reset since they do not persist past a restart.

In order for features to be activated experimentally using default branch prefs:

The feature must not start up before final-ui-startup is observed.

For instance, to set an observer:

Services.obs.addObserver(this, "final-ui-startup", true);

In this example, this would implement an observe(subject, topic, data) function which will be called when final-ui-startup is observed. See the Observer documentation for more information.

It must be possible to enable/disable the feature at runtime, via a pref change.

This is similar to the observer pattern above:

Services.prefs.addObserver("pref_name", this);

More information is available in the Preference service documentation.

Never use Services.prefs.prefHasUserValue(), or any other function specific to user branch prefs.
Prefs should be set by default in firefox.js

If your feature cannot abide by one or more of these rules (for instance, it needs to run at startup and/or cannot be toggled at runtime) then experimental preferences can be set on the user branch. This is more complex than using the methods described above; user branch prefs override the users choice, which is a really complex thing to try to support when flipping prefs experimentally. We also need to be careful to back up and reset the pref, and then figure out how to resolve conflicts if the user has changed the pref in the meantime.

Feature Gates

A new Feature Gate library for Firefox Desktop is now available.

Each feature gate should represent a different experimental treatment. If your experimental feature requires multiple flags, then Normandy will not be able to support this directly and an extension may be used.

Feature Gate caveats

The current Feature Gate library comes with a few caveats, and may not be appropriate for your situation:

Only JS is supported.
Always asynchronous.

Future versions of the Feature Gate API will include C++/Rust support and a synchronous API.

Using the Feature Gate library

Read the documentation to get started.

Extensions

Firefox currently supports the Web Extensions API.

If new WebExtension APIs are needed, they should land in-tree. Extensions which are signed by Mozilla can load privileged code using the WebExtension Experiments, but this is not preferred.

WebExtensions go through the same correctness and performance tests as other features. This is possible using the Mozilla tryserver by dropping your XPI into testing/profiles/common/extensions in mozilla-central and pushing to Tryserver - see the Testing Extensions section below.

NOTE - it is ideal to test against the version of Firefox which the extension will release against, but there is a bug related to artifact builds on release channels which must be worked around. The workaround is pretty simple (modify an artifacts.py file), but this bug being resolved will make it much simpler.

Each extension can represent a different experimental treatment (preferred), or the extension can choose the branch internally.

SHIELD studies

The previous version of the experiments program, SHIELD, always bundled privileged code with extensions and would do things such as mock UI features in Firefox.

This sort of approach is discouraged for new features - land these (or the necessary WebExtension APIs) in-tree instead.

For the moment, the SHIELD Study Add-on Utilities may be used if the extension needs to control the lifecycle of the study, but using one extension per experimental treatment makes this unnecessary and is preferred. The APIs provided by the SHIELD Study Add-on Utilities will be available as privileged APIs shipped with Firefox soon.

Development and Testing

Testing Built-in Features

Firefox features go through standard development and testing processes. See the Firefox developer guide for more information.

Testing Extensions

Extensions do not need to go through the same process, but should take advantage of Mozilla CI and bug tracking systems:

Use the Mozilla CI to test changes (tryserver).
Performance tests (this step is required) - extension XPI files should be placed in testing/profiles/common/extensions/, which will cause test harnesses to load the XPI.
Custom unit/functional tests (AKA xpcshell/mochitest) may be placed in testing/extensions, although running these tests outside Mozilla CI is acceptable so these are optional.
Receive reviewer approval. A Firefox peer must sign off if this extension contains privileged code, aka WebExtension Experiments.

Any Firefox Peer should be able to do the review, or point you to someone who can.

Extension is signed.
Email to pi-request@mozilla.com is sent to request QA
QA approval signed off in Bugzilla.
Extension is shipped via Normandy.

Example Extensions Testing Workflow

Note that for the below to work you only need Mercurial installed, but if you want to do local testing you must be set up to build Firefox. You don't need to build Firefox from source; artifact builds are sufficient.

In order to use Mozilla CI (AKA "Tryserver"), you must have a full clone of the mozilla-central repository:

hg clone https://hg.mozilla.org/mozilla-central
cd mozilla-central

Copy in unsigned XPI, and commit it to your local Mercurial repo:

cp ~/src/my-extension.xpi testing/profiles/common/extensions/
hg add testing/profiles/common/extensions/my-extension.xpi
hg commit -m "Bug nnn - Testing my extension" testing/profiles/common/extensions/my-extension.xpi

Push to Try:

./mach try -p linux64,macosx64,win64 -b do -u none -t all --artifact

This will run Mozilla CI tests on all platforms

Note that you must have Level 1 commit access to use tryserver. If you are interested in interacting with Mozilla CI from Github (which only requires users to be in the Mozilla GitHub org), check out the Taskcluster Integration proof-of-concept.

Also note that this requires an investment time to set up just as CircleCI or Travis-CI would, so it's not really appropriate for short-term projects. Use tryserver directly instead.

Mozilla Data Documentation