Jetstream datasets

• Statistics tables
  • Examples
• Client-window aggregate tables
• Enrollment tables
• Scheduling
• Code reference
• Documentation

Statistical summaries of telemetry data from experiments run in Mozilla products are provided by Jetstream. These summaries are published to BigQuery and serve both as the substrate for the result visualization platform and as a resource for data scientists.

Jetstream runs as part of the nightly ETL job (see Scheduling below). Jetstream is also run after pushes to the jetstream-config repository. Jetstream publishes tables to the dataset moz-fx-data-experiments.mozanalysis.

Experiments are analyzed using the concept of analysis windows. Analysis windows describe an interval marked from each client's day of enrollment. The "day 0" analysis window aggregates data from the days that each client enrolled in the experiment. Because the intervals are demarcated from enrollment, they are not calendar dates; for some clients in an experiment, day 0 could be a Tuesday, and for others a Saturday.

The week 0 analysis window aggregates data from each client's days 0 through 6, the week 1 window aggregates data from days 7 through 13, and so on.

Clients are given a fixed amount of time, specified in Experimenter and often a week long, to enroll. Final day 0 results are available for reporting at the end of the enrollment period, after the last eligible client has enrolled, and week 0 results are available a week after the enrollment period closes. Results for each window are published as soon as complete data is available for all enrolled clients.

The "overall" window, published after the experiment has ended, is a window beginning on each client's day 0 that spans the longest period for which all clients have complete data.

Jetstream computes statistics over several metrics by default, including for any features associated with the experiment in Experimenter. Data scientists can provide configuration to add additional metrics. Advice on configuring Jetstream can be found at the jetstream-config repository.

Statistics tables

The statistics tables contain statistical summaries of their corresponding aggregate tables. These tables are suitable for plotting directly without additional transformations.

Statistics tables are named like:

statistics_<slug>_{day, week, overall}_<index>

A view is also created that concatenates all statistics tables for an experiment of a given period type, named like:

statistics_<slug>_{daily, weekly, overall}

Statistics tables have the schema:

Each combination of (segment, metric, statistic, parameter, comparison, comparison_to_branch, ci_width, analysis_basis) uniquely describes a single data point.

The available segments in a table should be derived from inspection of the table.

Jetstream's Github wiki has a description of each statistic and comparison.

Examples

To extract the mean of active_hours for each branch from a weekly statistics view with a name like statistics_bug_12345_slug_weekly, you could run the query:

SELECT
    segment,
    window_index AS week,
    branch,
    point,
    lower,
    upper
FROM `moz-fx-data-experiments`.mozanalysis.statistics_bug_12345_slug_weekly
WHERE
    metric = "active_hours"
    AND statistic = "mean"
    AND comparison IS NULL

This query would return a row for each user segment, for each week of the experiment, for each branch, with the mean of the active_hours metric.

To see whether the absolute difference of the mean of active_hours was different between the control and treatment branches, you could run:

SELECT
    window_index AS week,
    branch,
    point,
    lower,
    upper
FROM `moz-fx-data-experiments`.mozanalysis.statistics_bug_12345_slug_weekly
WHERE
    metric = "active_hours"
    AND statistic = "mean"
    AND comparison = "difference"
    AND branch = "treatment"
    AND comparison_to_branch = "control"
    AND segment = "all"

This query would return a row for each week of the experiment containing an estimate of the absolute difference between the treatment and control branches for the segment containing all users.

Client-window aggregate tables

The aggregate tables contain one row per enrolled client_id. An aggregate table is written for each analysis window. The statistics tables are derived from the aggregate tables. The aggregate tables are less useful without additional processing but they may be useful for diagnostics.

Aggregate tables are named like:

<slug>_<analysis_basis>_{day,week,overall}_<index>

Aggregate tables have flexible schemas. Every table contains the columns:

The combination of (client_id, branch) is unique.

Each metric associated with the experiment defines an additional (arbitrarily-typed) column.

Each data source associated with the experiment defines additional <data_source>_has_contradictory_branch and <data_source>_has_non_enrolled_data columns, which respectively indicate whether client_id reported data from more than one branch or without any tagged branch in that dataset over that analysis window.

Each segment associated with the experiment defines an additional boolean column.

Enrollment tables

Enrollment tables contain enrollment information per client_id for which an enroll event has been received. An enrollment table for a specific experiment is created once after the enrollment period has completed. The enrollment table is then re-used in sub-sequent analysis runs.

Enrollment tables are named like:

enrollments_<slug>

Enrollment tables have flexible schemas, but every table contains the columns:

The combination of (client_id, branch) is unique.

Each segment defines an additional non-NULL boolean column per segment which is set to true if the client is in the segment and false otherwise.

Scheduling

Jetstream is updated nightly by telemetry-airflow. It is invoked by the jetstream DAG.

Code reference

Jetstream's datasets are generated by invoking Jetstream. Data scientists can configure Jetstream or trigger a Jetstream invocation by interacting with the jetstream-config repository.

Documentation

Additional documentation about Jetstream can be found in the Jetstream Experimenter Docs.