Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/concepts/metric_hub.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Metric Hub is a [repository](https://github.com/mozilla/metric-hub) that contains metric, data source and segment definitions that have been
reviewed and can be seen as the source of truth.
Definitions that are part of Metric Hub can be referenced in configurations of other tooling as well, such as [Jetstream](https://experimenter.info/deep-dives/jetstream/overview) and [OpMon](https://docs.telemetry.mozilla.org/cookbooks/operational_monitoring.html?highlight=opmon#operational-monitoring-opmon).
Definitions that are part of Metric Hub can be referenced in configurations of other tooling as well, such as [Jetstream](https://experimenter.info/data-analysis/jetstream/overview) and [OpMon](https://docs.telemetry.mozilla.org/cookbooks/operational_monitoring.html?highlight=opmon#operational-monitoring-opmon).

Metric Hub is a "thin" metric layer that lies between the Data Warehouse and consumers, such as analysis tools.
The Data Warehouse contains the datasets and tables referenced in the metric definitions. Analysis tools can reference metrics that are defined in metric-hub.
Expand Down Expand Up @@ -256,7 +256,7 @@ Segment SQL snippets need to be boolean expressions to be valid.

All the definitions are automatically available in some of our tooling:

- [Jetstream](https://experimenter.info/deep-dives/jetstream/overview) - used for analyzing experiments
- [Jetstream](https://experimenter.info/data-analysis/jetstream/overview) - used for analyzing experiments
- [mozanalysis](https://github.com/mozilla/mozanalysis) - a Python library which standardizes how experiment data is analyzed at Mozilla
- [OpMon](https://docs.telemetry.mozilla.org/cookbooks/operational_monitoring.html) - a tool for monitoring operational metrics
- [bigquery-etl](https://github.com/mozilla/bigquery-etl) - used for writing ETL queries
Expand Down
18 changes: 17 additions & 1 deletion src/cookbooks/operational_monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Access to the Looker Operational Monitoring dashboards is currently limited to M
## Configuring a Operational Monitoring project

To add or update a project configuration, open a pull request against the `opmon/` directory in [metric-hub](https://github.com/mozilla/metric-hub/tree/main/opmon).
Consider using and adding metric definitions to the [metric-hub](https://github.com/mozilla/metric-hub) that should be available for use across other tools, such as [Jetstream](https://experimenter.info/deep-dives/jetstream/overview), as well.
Consider using and adding metric definitions to the [metric-hub](https://github.com/mozilla/metric-hub) that should be available for use across other tools, such as [Jetstream](https://experimenter.info/data-analysis/jetstream/overview), as well.

CI checks will validate the columns, data sources, and SQL syntax. Once CI completes, the pull request gets automatically approved and can be merged. Results for the new project will be available within the next 24 hours.

Expand Down Expand Up @@ -460,6 +460,22 @@ The table `projects_v1` in `operational_monitoring_derived` contains metadata ab

OpMon is updated nightly by telemetry-airflow. It is invoked by the [operational_monitoring DAG](https://github.com/mozilla/telemetry-airflow/blob/main/dags/operational_monitoring.py).

## Common Use Cases

- **Rollout Monitoring:** Monitor rollouts to detect regressions before they affect the entire population. Compares enabled vs disabled branches for search metrics, performance, and error rates. The rollout monitoring dashboards get created automatically.
- **Build-over-Build Performance:** Detect performance regressions between consecutive builds on pre-release channels (Beta, Nightly). Tracks page load, FCP, input latency, JS execution, and memory metrics.
- **Product Health Dashboards:** Continuous monitoring of stability metrics without end dates: crash rates, hangs, exceptions, success/failure rates.
- **DAU/MAU Tracking and other Business Metrics:** Aggregate-level monitoring of daily/monthly active users, often with KPI forecasting to detect retention issues. Track revenue-impacting metrics like search monetization, subscription engagement, and customer satisfaction.
- **Experiments & A/B Tests:** As an addition to Experimenter, OpMon can be used to track experiment-specific success criteria and side effects for earlier signals.
- **Background Processes:** Monitor Firefox's background services (Background Update, Default Browser Agent). Tracks success rates, exception volumes, and timing metrics, often segmented by OS version.

## When to use OpMon?

- OpMon → Continuous monitoring, rollouts, build-over-build comparisons
- Experimenter → A/B tests requiring statistical significance testing
- GLAM → Ad-hoc histogram exploration
- Custom SQL → Complex queries beyond OpMon's aggregation model

## Experiments vs OpMon

The requirements for Operational Monitoring are related to, but mostly distinct from those for experiments:
Expand Down
2 changes: 1 addition & 1 deletion src/datasets/jetstream.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,4 +211,4 @@ by interacting with the [`jetstream-config`] repository.

## Documentation

Additional documentation about Jetstream can be found in the [Jetstream Experimenter Docs](https://experimenter.info/deep-dives/jetstream/overview).
Additional documentation about Jetstream can be found in the [Jetstream Experimenter Docs](https://experimenter.info/data-analysis/jetstream/overview).