Raw Ping Data

Introduction

We receive data from our users via pings. There are several types of pings, each containing different measurements and sent for different purposes. To review a complete list of ping types and their schemata, see this section of the Mozilla Source Tree Docs.

Many pings are also described by a JSONSchema specification which can be found in this repository.

There are a few pings that are central to delivering our core data collection primitives (Histograms, Events, Scalars) and for keeping an eye on Firefox behaviour (Environment, New Profiles, Updates, Crashes).

For instance, a user's first session in Firefox might have four pings like this:

"main" ping

The "main" ping is the workhorse of the Firefox Telemetry system. It delivers the Telemetry Environment as well as Histograms and Scalars for all process types that collect data in Firefox. It has several variants each with specific delivery characteristics:

Reason Sent when Notes
shutdown Firefox session ends cleanly Accounts for about 80% of all "main" pings. Sent by Pingsender immediately after Firefox shuts down, subject to conditions: Firefox 55+, if the OS isn't also shutting down, and if this isn't the client's first session. If Pingsender fails or isn't used, the ping is sent by Firefox at the beginning of the next Firefox session.
daily It has been more than 24 hours since the last "main" ping, and it is around local midnight In long-lived Firefox sessions we might go days without receiving a "shutdown" ping. Thus the "daily" ping is sent to ensure we occasionally hear from long-lived sessions.
environment-change Telemetry Environment changes Is sent immediately when triggered by Firefox (Installing or removing an addon or changing a monitored user preference are common ways for the Telemetry Environment to change)
aborted-session Firefox session doesn't end cleanly Sent by Firefox at the beginning of the next Firefox session.

It was introduced in Firefox 38.

"first-shutdown" ping

The "first-shutdown" ping is identical to the "main" ping with reason "shutdown" created at the end of the user's first session, but sent with a different ping type. This was introduced when we started using Pingsender to send shutdown pings as there would be a lot of first-session "shutdown" pings that we'd start receiving all of a sudden.

It is sent using Pingsender.

It was introduced in Firefox 57.

"event" ping

The "event" ping provides low-latency eventing support to Firefox Telemetry. It delivers the Telemetry Environment, Telemetry Events from all Firefox processes, and some diagnostic information about Event Telemetry. It is sent every hour if there have been events recorded, and up to once every 10 minutes (governed by a preference) if the maximum event limit for the ping (default to 1000 per process, governed by a preference) is reached before the hour is up.

It was introduced in Firefox 62.

"update" ping

Firefox Update is the most important means we have of reaching our users with the latest fixes and features. The "update" ping notifies us when an update is downloaded and ready to be applied (reason: "ready") and when the update has been successfully applied (reason: "success"). It contains the Telemetry Environment and information about the update.

It was introduced in Firefox 56.

"new-profile" ping

When a user starts up Firefox for the first time, a profile is created. Telemetry marks the occasion with the "new-profile" ping which sends the Telemetry Environment. It is sent either 30 minutes after Firefox starts running for the first time in this profile (reason: "startup") or at the end of the profile's first session (reason: "shutdown"), whichever comes first. "new-profile" pings are sent immediately when triggered. Those with reason "startup" are sent by Firefox. Those with reason "shutdown" are sent by Pingsender.

It was introduced in Firefox 55.

"crash" ping

The "crash" ping provides diagnostic information whenever a Firefox process exits abnormally. Unlike the "main" ping with reason "aborted-session", this ping does not contain Histograms or Scalars. It contains a Telemetry Environment, Crash Annotations, and Stack Traces.

It was introduced in Firefox 40.

"optout" ping

In the event a user opts out of Telemetry, we send one final "optout" ping to let us know. We try exactly once to send it, discarding the ping if sending fails. It contains only the common ping data and an empty payload.

It was introduced in Firefox 63.

Pingsender

Pingsender is a small application shipped with Firefox which attempts to send pings even if Firefox is not running. If Firefox has crashed or has already shut down we would otherwise have to wait for the next Firefox session to begin to send pings.

Pingsender was introduced in Firefox 54 to send "crash" pings. It was expanded to send "main" pings of reason "shutdown" in Firefox 55 (excepting the first session). It sends the "first-shutdown" ping since its introduction in Firefox 57.

Analysis

The large majority of analyses can be completed using only the main ping. This ping includes histograms, scalars, and other performance and diagnostic data.

Few analyses actually rely directly on any raw ping data. Instead, we provide derived datasets which are processed versions of these data, made to be:

  • Easier and faster to query
  • Organized to make the data easier to analyze
  • Cleaned of erroneous or misleading data

Before analyzing raw ping data, check to make sure there isn't already a derived dataset made for your purpose. If you do need to work with raw ping data, be aware that loading the data can take a while. Try to limit the size of your data by controlling the date range, etc.

Accessing the Data

You can access raw ping data from an ATMO cluster using the Dataset API. Raw ping data are not available in re:dash.

Further Reading

You can find the complete ping documentation. To augment our data collection, see Collecting New Data and the Data Collection Policy.

Data Reference

You can find the reference documentation for all ping types here.

Ping Metadata

The telemetry data pipeline appends metadata to arriving pings containing information about the ingestion environment, including timestamps; Geo-IP data about the client; and fields extracted from the ping or client headers that are useful for downstream processing.

The Dataset API represents this metadata by appending a meta key to the ping body. These fields may also be available as members of a metadata struct column in direct-to-parquet datasets.

Since the metadata are not present in the ping as it is sent by the client, these fields are documented here, instead of in the source tree docs.

As of September 28, 2018, members of the meta key on main pings include:

Key Description
appBuildId
appName e.g. "Firefox"
appUpdateChannel Raw incoming update channel. E.g. nightly-cck-example
appVendor e.g. "Mozilla"
appVersion
clientId
creationTimestamp Client creationDate field, transformed to nanoseconds since epoch
Date Client HTTP header reflecting the client time when the ping is sent, like Fri, 28 Sep 2018 14:01:57 GMT
DNT Client "do not track" HTTP header. Not present in all pings
docType e.g. "main"
documentId A UUID identifying the ping, generated by the client
geoCity from Geo-IP lookup of client IP; ?? if unknown
geoCountry from Geo-IP lookup of client IP; ?? if unknown
geoSubdivision1 from Geo-IP lookup of client IP; not present in all pings
geoSubdivision2 from Geo-IP lookup of client IP; not present in all pings
Host Public hostname of the ingestion endpoint
Hostname Private hostname of the ingestion endpoint
normalizedChannel Normalized update channel. E.g. "release"
normalizedOSVersion
os
reason Documented in the source tree
sampleId crc32(clientId) % 100
sourceName e.g. "telemetry"
sourceVersion Client version field (reflecting the version of the ping format)
submissionDate Server date (GMT) when ping was received, like 20180928; derived from Timestamp
telemetryEnabled Extracted value of environment.settings.telemetryEnabled, describing whether opt-in telemetry is enabled
Timestamp Server timestamp when the ping is received, expressed as nanoseconds since epoch
Type e.g. "telemetry"
X-PingSender-Version Present when a ping is sent with PingSender