Got some new data you want to send to us? How in the world do you send a new ping? Follow this guide to find out.
Do not try and implement new pings unless you know specifically what questions you're trying to answer. General questions about "How do users use our product?" won't cut it - these need to be specific, concrete asks that can be translated to data points. This will also make it easier down the line as you start data review.
More detail on how to design and implement new pings for Firefox Desktop can be found here.
For new telemetry pings, the namespace is simply
telemetry. For non-Telemetry
pings, choose a namespace that uniquely identifies the product that will be
generating the data.
The DocType is used to differentiate pings within a namespace. It can be as
event, but should generally be descriptive of the data being
Use JSON Schema to start with. See the "Adding a new schema" documentation and examples schemas in the Mozilla Pipeline Schemas repo. This schema is just used to validate the incoming data; any ping that doesn't match the schema will be removed. Validate your JSON Schema using a validation tool.
We already have automatic deduplicating based on
docId, which catches about 90% of duplicates and
removes them from the dataset.
The first schema added should be the JSON Schema made in step 2. Add at least one example ping which the data can be validated against. These test pings will be validated automatically during the build.
a Parquet output
schema should be added. This would add a new dataset, available in Re:dash.
The best documentation we have for the Parquet schema is by looking at the examples in
Parquet output also has a
metadata section. These are fields added to the ping at ingestion time;
they might come from the URL submitted to the edge server, or the IP Address used to make the request.
lists available metadata fields for all pings.
The stream you're interested in is probably
For example, look at
system-addon-deployment-diagnostics immediately under the
schema element has top-level fields (e.g.
Type), as well as more fields
Fields element. Any of these can be used in the
metadata section of your parquet schema,
Some common ones for Telemetry data might be:
And for non-Telemetry data:
Important Note: Schema evolution of nested structs is currently broken, so you will not be able to add
any fields in the future to your
metadata section. We recommend adding any that may seem useful.
For new data, use the edge validator to test your schema.
If your data is already being sent, and you want to test the schema you're writing on the data
that is currently being ingested, you can test your Parquet output in
Hindsight by using an output plugin.
See Core ping output plugin
for an example, where the parquet schema is specified as
parquet_schema. If no errors arise, that
means it should be correct. The "Deploy" button should not be used to actually deploy, that will be
done by operations in the next step.
Real-time analysis will be key to ensuring your data is being processed and parsed correctly.
It should follow the format specified in
This allows you to check validation errors, size changes, duplicates, and more. Once you have
the numbers set, file a
bug to let ops deploy it.
First confirm with the reviewers of your schema pull request that your schemas have been deployed.
In the following links, replace
appropriate values. Also replace
<namespace> if your
Once you've sent some pings, refer to the following real-time analysis plugins to verify that your data is being processed:
If this first graph shows ingestion errors, you can view the corresponding error messages in the third link. Otherwise, you should be able to view the last ten processed submissions via the second link. You can also write your own custom real-time analysis plugins using this same infrastructure if you desire; use the above plugins as examples and see here for a more detailed explanation.
If you encounter schema validation errors, you can fix your data or
submit another pull request
to amend your schemas. Backwards-incompatible schema changes should generally
be accompanied by an increment to
Once you've established that your pings are flowing through the real-time system, verify that you can access the data from the downstream systems.
In the Athena data source, a new table
<namespace>_<doctype>_parquet_<docversion> will be created for your data. A
<namespace>_<doctype>_parquet will also refer to the latest
docversion of the ping. The data is partitioned by
submission_date_s3 which is formatted as
20180130, and is
generally updated hourly. Refer to the STMO documentation
for general information about using Re:dash.
This table may take up to a day to appear in the Athena source; if you still don't see a table for your new ping after 24 hours, contact Data Operations so that they can investigate. Once the table is available, it should contain all the pings sent during that first day, regardless of how long it takes for the table to appear.
The data should be available in S3 at:
<namespace> should not be escaped.
Refer to the Spark FAQ for details on accessing this table via ATMO.
You can schedule it on Airflow, or you can run it as a job in ATMO. If the output is parquet, you can add it to the Hive metastore to have it available in re:dash. Check the docs on creating your own datasets.
Last steps! What are you using this data for anyway?