feat: build search_clients_daily_v9

[kbammarito]

Description

This PR builds search_clients_daily_v9 from Glean (migrating from legacy telemetry).

Related Tickets & Documents

• DENG-8178

Reviewer, please follow this checklist

[Standard8]

Thank you for working on this. It is great to see this moving along. Sorry for the delay in responding, there was lots of thinking and checking to do.

[Standard8]

I realise the current serp_events doesn't have it, but we could really do with partner_code here as well (can we get it added there?

This would allow for better breakdowns by partner code, something that we can't analyse today with the existing tables, yet we should really be taking it into account.

[Standard8]

We should also have something like search_engine as provider_id. Similar to the sap_source field, the user may be searching different providers which aren't triggered via the normal search access points. Hence we need to include those as well.

Though could we also change "duckduckgo" to "ddg" within the field so that it matches the provider_id field for SAP and the search_engine_default_provider_id for the default search engine?

[Standard8]

We might want to also add the normalized_engine field in here as well - but only filled in from search_engine (there's no name field for SERPs, since there's a limited amount.

This would make it easier to search for both tagged and organic SERPs using the normalized_engine field, which you might want to do if you're also including SAP data.

[kbammarito]

@Standard8: with your first comment, are we understanding correctly that you want partner_code added to the serp_events table itself (not just us pulling it in through another CTE)?

[Standard8]

Sorry, I realise I wasn't too clear here. Yes, the provider and partner_code data need pulling in from the serp.impression event (from serp_events). They're dedicated to that event, you can't pull them from the sap.counts event, since they're different events with potentially different totals per provider/partner code.

So, in summary, I think this serp_events cte needs the following adding:

CASE
  WHEN search_engine = "duckduckgo" then "ddg"
  ELSE search_engine
END as provider_id,
partner_code,

(This will need partner_code also adding to the mozdata.firefox_desktop.serp_events table...).

[Standard8]

As noted above, provider_id, partner_code, search_access_point should be from both sap & serp.

[kbammarito]

@Standard8: we don't see the provider_id and partner_code fields in the serp_events table. Do you want us to pull them in from serp_impressions ping (it has provider and partner_code)? Or are you saying they should be added to the serp_events table?

[Standard8]

I believe provider_id (aka provider) is in the serp_events table as search_engine.

I think the partner_code should also be added to serp_events table if possible - That's an important aspect that we're currently missing.

[betling]

@Standard8 DE can add partner_code to the serp_events table. That is not a big lift. Does it need to be backfilled? I understand that will take a much longer time if needed but that they could add it to their backlog. @kbammarito or @alekhyamoz can correct me as needed.

[Standard8]

Presumably we can request backfill at a later time? If so, I'd suggest leaving it for now, and if we need a bit more history when we start using the new table, then we can request it.

[skahmann3]

I believe the plan for "backfill" was to create another versioned table. We'd do a small backfill to ensure metric trends haven't changed, but the final table will concatenate the two tables: v1 if < date and v2 if > date. Stakeholders will keep using legacy data for past results, but future results will be transitioned to the new metrics.

[Standard8]

I've added a few responses. I'm assuming the "done" comments are ones that have been done locally, and not yet pushed to this PR.

[Standard8]

Sorry, I realise I wasn't too clear here. Yes, the provider and partner_code data need pulling in from the serp.impression event (from serp_events). They're dedicated to that event, you can't pull them from the sap.counts event, since they're different events with potentially different totals per provider/partner code.

So, in summary, I think this serp_events cte needs the following adding:

CASE
  WHEN search_engine = "duckduckgo" then "ddg"
  ELSE search_engine
END as provider_id,
partner_code,

(This will need partner_code also adding to the mozdata.firefox_desktop.serp_events table...).

[Standard8]

I believe provider_id (aka provider) is in the serp_events table as search_engine.

I think the partner_code should also be added to serp_events table if possible - That's an important aspect that we're currently missing.

[dataops-ci-bot]

Integration report for "Merge branch 'DENG-8178-build-new-glean-based-equivalent-table-to-replace-search_clients_daily_v8' of github.com:mozilla/bigquery-etl into DENG-8178-build-new-glean-based-equivalent-table-to-replace-search_clients_daily_v8"

sql.diff

Click to expand!

diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_glean_usage.py /tmp/workspace/generated-sql/dags/bqetl_glean_usage.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_glean_usage.py	2025-11-25 01:19:37.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_glean_usage.py	2025-11-25 01:18:15.000000000 +0000
@@ -3930,6 +3930,13 @@
         )
 
         ExternalTaskMarker(
+            task_id="bqetl_search__wait_for_firefox_desktop_derived__events_stream__v1",
+            external_dag_id="bqetl_search",
+            external_task_id="wait_for_firefox_desktop_derived__events_stream__v1",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=82800)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
             task_id="bqetl_ech_adoption_rate__wait_for_firefox_desktop_derived__events_stream__v1",
             external_dag_id="bqetl_ech_adoption_rate",
             external_task_id="wait_for_firefox_desktop_derived__events_stream__v1",
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_search.py /tmp/workspace/generated-sql/dags/bqetl_search.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_search.py	2025-11-25 01:19:30.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_search.py	2025-11-25 01:18:02.000000000 +0000
@@ -105,6 +105,45 @@
         pool="DATA_ENG_EXTERNALTASKSENSOR",
     )
 
+    wait_for_copy_deduplicate_all = ExternalTaskSensor(
+        task_id="wait_for_copy_deduplicate_all",
+        external_dag_id="copy_deduplicate",
+        external_task_id="copy_deduplicate_all",
+        execution_delta=datetime.timedelta(seconds=7200),
+        check_existence=True,
+        mode="reschedule",
+        poke_interval=datetime.timedelta(minutes=5),
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
+    wait_for_firefox_desktop_derived__events_stream__v1 = ExternalTaskSensor(
+        task_id="wait_for_firefox_desktop_derived__events_stream__v1",
+        external_dag_id="bqetl_glean_usage",
+        external_task_id="firefox_desktop.firefox_desktop_derived__events_stream__v1",
+        execution_delta=datetime.timedelta(seconds=3600),
+        check_existence=True,
+        mode="reschedule",
+        poke_interval=datetime.timedelta(minutes=5),
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
+    wait_for_firefox_desktop_serp_events__v2 = ExternalTaskSensor(
+        task_id="wait_for_firefox_desktop_serp_events__v2",
+        external_dag_id="bqetl_serp",
+        external_task_id="firefox_desktop_serp_events__v2",
+        execution_delta=datetime.timedelta(seconds=10800),
+        check_existence=True,
+        mode="reschedule",
+        poke_interval=datetime.timedelta(minutes=5),
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
     wait_for_clients_first_seen_v3 = ExternalTaskSensor(
         task_id="wait_for_clients_first_seen_v3",
         external_dag_id="bqetl_analytics_tables",
@@ -233,6 +272,44 @@
             search_derived__search_clients_daily__v8
         )
 
+    search_derived__search_clients_daily__v9 = bigquery_etl_query(
+        task_id="search_derived__search_clients_daily__v9",
+        destination_table="search_clients_daily_v9",
+        dataset_id="search_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="akommasani@mozilla.com",
+        email=[
+            "akomar@mozilla.com",
+            "akommasani@mozilla.com",
+            "cmorales@mozilla.com",
+            "kbammarito@mozilla.com",
+            "telemetry-alerts@mozilla.com",
+        ],
+        date_partition_parameter="submission_date",
+        depends_on_past=False,
+    )
+
+    with TaskGroup(
+        "search_derived__search_clients_daily__v9_external",
+    ) as search_derived__search_clients_daily__v9_external:
+        ExternalTaskMarker(
+            task_id="jetstream__wait_for_search_clients_daily",
+            external_dag_id="jetstream",
+            external_task_id="wait_for_search_clients_daily",
+            execution_date="{{ (execution_date + macros.timedelta(seconds=3600)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
+            task_id="operational_monitoring__wait_for_search_clients_daily",
+            external_dag_id="operational_monitoring",
+            external_task_id="wait_for_search_clients_daily",
+            execution_date="{{ (execution_date + macros.timedelta(seconds=3600)).isoformat() }}",
+        )
+
+        search_derived__search_clients_daily__v9_external.set_upstream(
+            search_derived__search_clients_daily__v9
+        )
+
     search_derived__search_clients_last_seen__v1 = bigquery_etl_query(
         task_id="search_derived__search_clients_last_seen__v1",
         destination_table="search_clients_last_seen_v1",
@@ -336,6 +413,20 @@
         wait_for_telemetry_derived__clients_daily_joined__v1
     )
 
+    search_derived__search_clients_daily__v9.set_upstream(wait_for_copy_deduplicate_all)
+
+    search_derived__search_clients_daily__v9.set_upstream(
+        wait_for_firefox_desktop_derived__events_stream__v1
+    )
+
+    search_derived__search_clients_daily__v9.set_upstream(
+        wait_for_firefox_desktop_serp_events__v2
+    )
+
+    search_derived__search_clients_daily__v9.set_upstream(
+        wait_for_revenue_derived__monetization_blocking_addons__v2
+    )
+
     search_derived__search_clients_last_seen__v1.set_upstream(
         search_derived__search_clients_daily__v8
     )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_serp.py /tmp/workspace/generated-sql/dags/bqetl_serp.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_serp.py	2025-11-25 01:19:31.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_serp.py	2025-11-25 01:18:04.000000000 +0000
@@ -88,6 +88,13 @@
         "firefox_desktop_serp_events__v2_external",
     ) as firefox_desktop_serp_events__v2_external:
         ExternalTaskMarker(
+            task_id="bqetl_search__wait_for_firefox_desktop_serp_events__v2",
+            external_dag_id="bqetl_search",
+            external_task_id="wait_for_firefox_desktop_serp_events__v2",
+            execution_date="{{ (execution_date - macros.timedelta(days=-1, seconds=75600)).isoformat() }}",
+        )
+
+        ExternalTaskMarker(
             task_id="bqetl_search_dashboard__wait_for_firefox_desktop_serp_events__v2",
             external_dag_id="bqetl_search_dashboard",
             external_task_id="wait_for_firefox_desktop_serp_events__v2",
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived: search_clients_daily_v9
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/01_adblocker.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/01_adblocker.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/01_adblocker.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/01_adblocker.sql	2025-11-25 01:10:52.000000000 +0000
@@ -0,0 +1,40 @@
+WITH
+-- list of ad blocking addons produced using this logic: https://github.com/mozilla/search-adhoc-analysis/tree/master/monetization-blocking-addons
+adblocker_addons_cte AS (
+  SELECT
+    addon_id,
+    addon_name
+  FROM
+    `moz-fx-data-shared-prod.revenue.monetization_blocking_addons`
+  WHERE
+    blocks_monetization
+),
+-- this table is the new glean adblocker addons metric ping (used to be legacy telemetry)
+clients_with_adblocker_addons_cte AS (
+  SELECT
+    client_info.client_id,
+    DATE(submission_timestamp) AS submission_date,
+    TRUE AS has_adblocker_addon
+  FROM
+    `moz-fx-data-shared-prod.firefox_desktop_stable.metrics_v1`,
+    UNNEST(JSON_QUERY_ARRAY(metrics.object.addons_active_addons)) AS addons
+  INNER JOIN
+    adblocker_addons_cte
+    ON adblocker_addons_cte.addon_id = JSON_VALUE(addons, '$.id')
+  WHERE
+    DATE(submission_timestamp)
+    BETWEEN '2025-06-25'
+    AND '2025-09-25'
+    AND sample_id = 0
+    -- date(submission_timestamp) = @submission_date
+    AND NOT BOOL(JSON_QUERY(addons, '$.userDisabled'))
+    AND NOT BOOL(JSON_QUERY(addons, '$.appDisabled'))
+    AND NOT BOOL(JSON_QUERY(addons, '$.blocklisted'))
+  GROUP BY
+    client_id,
+    DATE(submission_timestamp)
+)
+SELECT
+  *
+FROM
+  clients_with_adblocker_addons_cte
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/02_sap_is_enterprise.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/02_sap_is_enterprise.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/02_sap_is_enterprise.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/02_sap_is_enterprise.sql	2025-11-25 01:10:52.000000000 +0000
@@ -0,0 +1,24 @@
+-- sap_is_enterprise_cte
+-- we still need this because of document_id is not null
+SELECT
+  client_id,
+  DATE(submission_timestamp) AS submission_date,
+  CAST(
+    JSON_VALUE(
+      array_last(ARRAY_AGG(metrics.boolean.policies_is_enterprise ORDER BY event_timestamp DESC)),
+      '$'
+    ) AS boolean
+  ) AS policies_is_enterprise
+FROM
+  `moz-fx-data-shared-prod.firefox_desktop_derived.events_stream_v1`
+WHERE
+  DATE(submission_timestamp)
+  BETWEEN '2025-06-25'
+  AND '2025-09-25'
+  AND sample_id = 0
+-- date(submission_timestamp) = @submission_date
+  AND event = 'sap.counts'
+  AND document_id IS NOT NULL
+GROUP BY
+  client_id,
+  DATE(submission_timestamp)
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/03_sap_events_with_client_info.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/03_sap_events_with_client_info.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/03_sap_events_with_client_info.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/03_sap_events_with_client_info.sql	2025-11-25 01:10:52.000000000 +0000
@@ -0,0 +1,140 @@
+CREATE TEMP FUNCTION safe_parse_timestamp(ts string) AS (
+  COALESCE(
+        -- full datetime with offset
+    SAFE.PARSE_TIMESTAMP("%F%T%Ez", ts),
+        -- date + offset (no time)
+    SAFE.PARSE_TIMESTAMP("%F%Ez", ts),
+        -- datetime with space before offset
+    SAFE.PARSE_TIMESTAMP("%F%T%Ez", REGEXP_REPLACE(ts, r"(\+|\-)(\d{2}):(\d{2})", "\\1\\2\\3"))
+  )
+);
+
+-- sap_events_with_client_info_cte
+SELECT
+  client_id AS client_id,
+  DATE(submission_timestamp) AS submission_date,
+  CASE
+    WHEN JSON_VALUE(event_extra.provider_id) = 'other'
+      THEN `moz-fx-data-shared-prod.udf.normalize_search_engine`(
+          JSON_VALUE(event_extra.provider_name)
+        )
+    ELSE `moz-fx-data-shared-prod.udf.normalize_search_engine`(JSON_VALUE(event_extra.provider_id))
+  END AS normalized_engine, -- this is "engine" in v8
+  CASE
+    WHEN JSON_VALUE(event_extra.partner_code) = ''
+      THEN NULL
+    ELSE JSON_VALUE(event_extra.partner_code)
+  END AS partner_code,
+  CASE
+    WHEN JSON_VALUE(event_extra.source) = 'urlbar-handoff'
+      THEN 'urlbar_handoff'
+    ELSE JSON_VALUE(event_extra.source)
+  END AS source,
+-- end as search_access_point, -- rename
+  sample_id,
+  profile_group_id,
+  legacy_telemetry_client_id, -- adding this for now so people can join to it if needed
+  normalized_country_code AS country,
+  normalized_app_name AS app_version,
+  client_info.app_channel AS channel,
+  normalized_channel,
+  client_info.locale,
+  client_info.os,
+  normalized_os,
+  client_info.os_version,
+  normalized_os_version,
+  client_info.windows_build_number,
+  client_info.distribution.name AS distribution_id,
+  UNIX_DATE(DATE(safe_parse_timestamp(client_info.first_run_date))) AS profile_creation_date,
+  CAST(JSON_VALUE(metrics.string.region_home_region, '$') AS string) AS region_home_region,
+  CAST(
+    JSON_VALUE(metrics.boolean.usage_is_default_browser, '$') AS boolean
+  ) AS usage_is_default_browser,
+  CAST(
+    JSON_VALUE(metrics.string.search_engine_default_display_name, '$') AS string
+  ) AS search_engine_default_display_name,
+  CAST(
+    JSON_VALUE(metrics.string.search_engine_default_load_path, '$') AS string
+  ) AS search_engine_default_load_path,
+  CAST(
+    JSON_VALUE(metrics.string.search_engine_default_partner_code, '$') AS string
+  ) AS search_engine_default_partner_code,
+  CAST(
+    JSON_VALUE(metrics.string.search_engine_default_provider_id, '$') AS string
+  ) AS search_engine_default_provider_id,
+  CAST(
+    JSON_VALUE(metrics.url.search_engine_default_submission_url, '$') AS string
+  ) AS search_engine_default_submission_url,
+  CAST(
+    JSON_VALUE(metrics.boolean.search_engine_default_overridden_by_third_party, '$') AS boolean
+  ) AS search_engine_default_overridden_by_third_party,
+  CAST(
+    JSON_VALUE(metrics.string.search_engine_private_display_name, '$') AS string
+  ) AS search_engine_private_display_name,
+  CAST(
+    JSON_VALUE(metrics.string.search_engine_private_load_path, '$') AS string
+  ) AS search_engine_private_load_path,
+  CAST(
+    JSON_VALUE(metrics.string.search_engine_private_partner_code, '$') AS string
+  ) AS search_engine_private_partner_code,
+  CAST(
+    JSON_VALUE(metrics.string.search_engine_private_provider_id, '$') AS string
+  ) AS search_engine_private_provider_id,
+  CAST(
+    JSON_VALUE(metrics.url.search_engine_private_submission_url, '$') AS string
+  ) AS search_engine_private_submission_url,
+  CAST(
+    JSON_VALUE(metrics.boolean.search_engine_private_overridden_by_third_party, '$') AS boolean
+  ) AS search_engine_private_overridden_by_third_party,
+  JSON_VALUE(event_extra.provider_name) AS provider_name,
+  JSON_VALUE(event_extra.provider_id) AS provider_id,
+  CAST(
+    JSON_VALUE(event_extra.overridden_by_third_party, '$') AS boolean
+  ) AS overridden_by_third_party,
+  ping_info.start_time AS subsession_start_time,
+  ping_info.end_time AS subsession_end_time,
+  ping_info.seq AS subsession_counter,
+  [
+    STRUCT(
+      json_keys(experiments)[OFFSET(0)] AS key,
+      STRUCT(
+        REPLACE(
+          TO_JSON_STRING(experiments[json_keys(experiments)[OFFSET(0)]].branch),
+          '"',
+          ''
+        ) AS branch,
+        STRUCT(
+          REPLACE(
+            TO_JSON_STRING(experiments[json_keys(experiments)[OFFSET(0)]].extra.type),
+            '"',
+            ''
+          ) AS type,
+          REPLACE(
+            TO_JSON_STRING(experiments[json_keys(experiments)[OFFSET(0)]].extra.enrollment_id),
+            '"',
+            ''
+          ) AS enrollment_id
+        ) AS extra
+      ) AS value
+    )
+  ] AS experiments
+FROM
+  `moz-fx-data-shared-prod.firefox_desktop_derived.events_stream_v1`
+WHERE
+  DATE(submission_timestamp)
+  BETWEEN '2025-06-25'
+  AND '2025-09-25'
+  AND sample_id = 0
+-- date(submission_timestamp) = @submission_date
+  AND event = 'sap.counts'
+-- this is to get the last instance
+QUALIFY
+  ROW_NUMBER() OVER (
+    PARTITION BY
+      client_id,
+      submission_date,
+      normalized_engine,
+      source
+    ORDER BY
+      event_timestamp DESC
+  ) = 1
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/04_sap_events_clients_ad_enterprise.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/04_sap_events_clients_ad_enterprise.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/04_sap_events_clients_ad_enterprise.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/04_sap_events_clients_ad_enterprise.sql	2025-11-25 01:10:52.000000000 +0000
@@ -0,0 +1,54 @@
+-- sap_events_clients_ad_enterprise_cte
+SELECT
+  sap_events_with_client_info_cte.client_id,
+  sap_events_with_client_info_cte.submission_date,
+  sap_events_with_client_info_cte.normalized_engine,
+  sap_events_with_client_info_cte.partner_code,
+  sap_events_with_client_info_cte.source,
+-- sap_events_with_client_info_cte.source as search_access_point, -- rename
+  sap_events_with_client_info_cte.sample_id,
+  sap_events_with_client_info_cte.profile_group_id,
+  sap_events_with_client_info_cte.legacy_telemetry_client_id,
+  sap_events_with_client_info_cte.country,
+  sap_events_with_client_info_cte.app_version,
+  sap_events_with_client_info_cte.channel,
+  sap_events_with_client_info_cte.normalized_channel,
+  sap_events_with_client_info_cte.locale,
+  sap_events_with_client_info_cte.os,
+  sap_events_with_client_info_cte.normalized_os,
+  sap_events_with_client_info_cte.os_version,
+  sap_events_with_client_info_cte.normalized_os_version,
+  sap_events_with_client_info_cte.windows_build_number,
+  sap_events_with_client_info_cte.distribution_id,
+  sap_events_with_client_info_cte.profile_creation_date,
+  sap_events_with_client_info_cte.region_home_region,
+  sap_events_with_client_info_cte.usage_is_default_browser,
+  sap_events_with_client_info_cte.search_engine_default_display_name,
+  sap_events_with_client_info_cte.search_engine_default_load_path,
+  sap_events_with_client_info_cte.search_engine_default_partner_code,
+  sap_events_with_client_info_cte.search_engine_default_provider_id,
+  sap_events_with_client_info_cte.search_engine_default_submission_url,
+  sap_events_with_client_info_cte.search_engine_default_overridden_by_third_party,
+  sap_events_with_client_info_cte.search_engine_private_display_name,
+  sap_events_with_client_info_cte.search_engine_private_load_path,
+  sap_events_with_client_info_cte.search_engine_private_partner_code,
+  sap_events_with_client_info_cte.search_engine_private_provider_id,
+  sap_events_with_client_info_cte.search_engine_private_submission_url,
+  sap_events_with_client_info_cte.search_engine_private_overridden_by_third_party,
+  sap_events_with_client_info_cte.provider_name,
+  sap_events_with_client_info_cte.provider_id,
+  sap_events_with_client_info_cte.overridden_by_third_party,
+  sap_events_with_client_info_cte.subsession_start_time,
+  sap_events_with_client_info_cte.subsession_end_time,
+  sap_events_with_client_info_cte.subsession_counter,
+  sap_events_with_client_info_cte.experiments,
+  clients_with_adblocker_addons_cte.has_adblocker_addon,
+  sap_is_enterprise_cte.policies_is_enterprise
+FROM
+  `search_derived.search_clients_daily_v9.sap_events_with_client_info_cte`
+LEFT JOIN
+  `search_derived.search_clients_daily_v9.clients_with_adblocker_addons_cte`
+  USING (client_id, submission_date)
+LEFT JOIN
+  `search_derived.search_clients_daily_v9.sap_is_enterprise_cte`
+  USING (client_id, submission_date)
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/05_sap_aggregates.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/05_sap_aggregates.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/05_sap_aggregates.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/05_sap_aggregates.sql	2025-11-25 01:10:52.000000000 +0000
@@ -0,0 +1,69 @@
+CREATE TEMP FUNCTION safe_parse_timestamp(ts string) AS (
+  COALESCE(
+        -- full datetime with offset
+    SAFE.PARSE_TIMESTAMP("%F%T%Ez", ts),
+        -- date + offset (no time)
+    SAFE.PARSE_TIMESTAMP("%F%Ez", ts),
+        -- datetime with space before offset
+    SAFE.PARSE_TIMESTAMP("%F%T%Ez", REGEXP_REPLACE(ts, r"(\+|\-)(\d{2}):(\d{2})", "\\1\\2\\3"))
+  )
+);
+
+-- sap_aggregates_cte
+SELECT
+  client_id,
+  DATE(submission_timestamp) AS submission_date,
+  CASE
+    WHEN JSON_VALUE(event_extra.provider_id) = 'other'
+      THEN `moz-fx-data-shared-prod.udf.normalize_search_engine`(
+          JSON_VALUE(event_extra.provider_name)
+        )
+    ELSE `moz-fx-data-shared-prod.udf.normalize_search_engine`(JSON_VALUE(event_extra.provider_id))
+  END AS normalized_engine,
+  CASE
+    WHEN JSON_VALUE(event_extra.source) = 'urlbar-handoff'
+      THEN 'urlbar_handoff'
+    ELSE JSON_VALUE(event_extra.source)
+  END AS source,
+-- end as search_access_point, -- rename
+  MAX(UNIX_DATE(DATE((ping_info.parsed_start_time)))) - MAX(
+    UNIX_DATE(DATE(safe_parse_timestamp(client_info.first_run_date)))
+  ) AS profile_age_in_days,
+  COUNT(*) AS sap,
+  COUNTIF(ping_info.seq = 1) AS sessions_started_on_this_day,
+  SUM(
+    CAST(JSON_EXTRACT_SCALAR(metrics.counter, '$.browser_engagement_active_ticks') AS float64) / (
+      3600 / 5
+    )
+  ) AS active_hours_sum,
+  SUM(
+    CAST(
+      JSON_EXTRACT_SCALAR(metrics.counter, '$.browser_engagement_tab_open_event_count') AS float64
+    )
+  ) AS scalar_parent_browser_engagement_tab_open_event_count_sum,
+  SUM(
+    CAST(JSON_EXTRACT_SCALAR(metrics.counter, '$.browser_engagement_uri_count') AS float64)
+  ) AS scalar_parent_browser_engagement_total_uri_count_sum,
+  MAX(
+    CAST(
+      JSON_EXTRACT_SCALAR(
+        metrics.quantity,
+        '$.browser_engagement_max_concurrent_tab_count'
+      ) AS float64
+    )
+  ) AS concurrent_tab_count_max
+FROM
+  `moz-fx-data-shared-prod.firefox_desktop_derived.events_stream_v1`
+WHERE
+  DATE(submission_timestamp)
+  BETWEEN '2025-06-25'
+  AND '2025-09-25'
+  AND sample_id = 0
+-- date(submission_timestamp) = @submission_date
+  AND event = 'sap.counts'
+GROUP BY
+  client_id,
+  submission_date,
+  normalized_engine,
+  source
+-- search_access_point
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/06_sap_final.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/06_sap_final.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/06_sap_final.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/06_sap_final.sql	2025-11-25 01:10:52.000000000 +0000
@@ -0,0 +1,59 @@
+-- final_sap_cte
+SELECT
+  sap_events_clients_ad_enterprise_cte.client_id,
+  sap_events_clients_ad_enterprise_cte.submission_date,
+  sap_events_clients_ad_enterprise_cte.normalized_engine,
+  sap_events_clients_ad_enterprise_cte.partner_code,
+  sap_events_clients_ad_enterprise_cte.source,
+-- sap_events_clients_ad_enterprise_cte.source as search_access_point, -- rename
+  sap_events_clients_ad_enterprise_cte.sample_id,
+  sap_events_clients_ad_enterprise_cte.profile_group_id,
+  sap_events_clients_ad_enterprise_cte.legacy_telemetry_client_id,
+  sap_events_clients_ad_enterprise_cte.country,
+  sap_events_clients_ad_enterprise_cte.app_version,
+  sap_events_clients_ad_enterprise_cte.channel,
+  sap_events_clients_ad_enterprise_cte.normalized_channel,
+  sap_events_clients_ad_enterprise_cte.locale,
+  sap_events_clients_ad_enterprise_cte.os,
+  sap_events_clients_ad_enterprise_cte.normalized_os,
+  sap_events_clients_ad_enterprise_cte.os_version,
+  sap_events_clients_ad_enterprise_cte.normalized_os_version,
+  sap_events_clients_ad_enterprise_cte.windows_build_number,
+  sap_events_clients_ad_enterprise_cte.distribution_id,
+  sap_events_clients_ad_enterprise_cte.profile_creation_date,
+  sap_events_clients_ad_enterprise_cte.region_home_region,
+  sap_events_clients_ad_enterprise_cte.usage_is_default_browser,
+  sap_events_clients_ad_enterprise_cte.search_engine_default_display_name,
+  sap_events_clients_ad_enterprise_cte.search_engine_default_load_path,
+  sap_events_clients_ad_enterprise_cte.search_engine_default_partner_code,
+  sap_events_clients_ad_enterprise_cte.search_engine_default_provider_id,
+  sap_events_clients_ad_enterprise_cte.search_engine_default_submission_url,
+  sap_events_clients_ad_enterprise_cte.search_engine_default_overridden_by_third_party,
+  sap_events_clients_ad_enterprise_cte.search_engine_private_display_name,
+  sap_events_clients_ad_enterprise_cte.search_engine_private_load_path,
+  sap_events_clients_ad_enterprise_cte.search_engine_private_partner_code,
+  sap_events_clients_ad_enterprise_cte.search_engine_private_provider_id,
+  sap_events_clients_ad_enterprise_cte.search_engine_private_submission_url,
+  sap_events_clients_ad_enterprise_cte.search_engine_private_overridden_by_third_party,
+  sap_events_clients_ad_enterprise_cte.provider_name,
+  sap_events_clients_ad_enterprise_cte.provider_id,
+  sap_events_clients_ad_enterprise_cte.overridden_by_third_party,
+  sap_events_clients_ad_enterprise_cte.subsession_start_time,
+  sap_events_clients_ad_enterprise_cte.subsession_end_time,
+  sap_events_clients_ad_enterprise_cte.subsession_counter,
+  sap_events_clients_ad_enterprise_cte.experiments,
+  sap_events_clients_ad_enterprise_cte.has_adblocker_addon,
+  sap_events_clients_ad_enterprise_cte.policies_is_enterprise,
+  sap_aggregates_cte.profile_age_in_days,
+  sap_aggregates_cte.sap,
+  sap_aggregates_cte.sessions_started_on_this_day,
+  sap_aggregates_cte.active_hours_sum,
+  sap_aggregates_cte.scalar_parent_browser_engagement_tab_open_event_count_sum,
+  sap_aggregates_cte.scalar_parent_browser_engagement_total_uri_count_sum,
+  sap_aggregates_cte.concurrent_tab_count_max
+FROM
+  `search_derived.search_clients_daily_v9.sap_events_clients_ad_enterprise_cte`
+LEFT JOIN
+  `search_derived.search_clients_daily_v9.sap_aggregates_cte`
+  USING (client_id, submission_date, normalized_engine, source)
+-- using(client_id, submission_date, normalized_engine, partner_code, search_access_point) -- rename
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/07_serp_is_enterprise.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/07_serp_is_enterprise.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/07_serp_is_enterprise.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/07_serp_is_enterprise.sql	2025-11-25 01:10:52.000000000 +0000
@@ -0,0 +1,20 @@
+-- serp_is_enterprise_cte
+-- we still need this because of document_id is not null
+SELECT
+  glean_client_id AS client_id,
+  submission_date,
+  array_last(
+    ARRAY_AGG(policies_is_enterprise ORDER BY event_timestamp DESC)
+  ) AS policies_is_enterprise
+FROM
+  `moz-fx-data-shared-prod.firefox_desktop_derived.serp_events_v2`
+WHERE
+  submission_date
+  BETWEEN '2025-06-25'
+  AND '2025-09-25'
+  AND sample_id = 0
+-- submission_date = @submission_date
+  AND document_id IS NOT NULL
+GROUP BY
+  client_id,
+  submission_date
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/08_serp_events_with_client_info.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/08_serp_events_with_client_info.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/08_serp_events_with_client_info.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/08_serp_events_with_client_info.sql	2025-11-25 01:10:52.000000000 +0000
@@ -0,0 +1,86 @@
+CREATE TEMP FUNCTION safe_parse_timestamp(ts string) AS (
+  COALESCE(
+        -- full datetime with offset
+    SAFE.PARSE_TIMESTAMP("%F%T%Ez", ts),
+        -- date + offset (no time)
+    SAFE.PARSE_TIMESTAMP("%F%Ez", ts),
+        -- datetime with space before offset
+    SAFE.PARSE_TIMESTAMP("%F%T%Ez", REGEXP_REPLACE(ts, r"(\+|\-)(\d{2}):(\d{2})", "\\1\\2\\3"))
+  )
+);
+
+-- serp_events_with_client_info_cte
+SELECT
+  glean_client_id AS client_id,
+  submission_date,
+  `moz-fx-data-shared-prod.udf.normalize_search_engine`(
+    search_engine
+  ) AS serp_provider_id, -- this is engine
+  CASE
+    WHEN partner_code = ''
+      THEN NULL
+    ELSE partner_code
+  END AS partner_code,
+  sap_source AS serp_search_access_point,
+  sample_id,
+  profile_group_id,
+  legacy_telemetry_client_id,
+  normalized_country_code AS country,
+  normalized_app_name AS app_version,
+  channel,
+  normalized_channel,
+  locale,
+  os,
+  normalized_os,
+  os_version,
+  normalized_os_version,
+  CASE
+    WHEN mozfun.norm.os(os) = "Windows"
+      THEN mozfun.norm.windows_version_info(os, os_version, windows_build_number)
+    ELSE CAST(mozfun.norm.truncate_version(os_version, "major") AS STRING)
+  END AS os_version_major,
+  CASE
+    WHEN mozfun.norm.os(os) = "Windows"
+      THEN mozfun.norm.windows_version_info(os, os_version, windows_build_number)
+    ELSE CAST(mozfun.norm.truncate_version(os_version, "minor") AS string)
+  END AS os_version_minor,
+  windows_build_number,
+  distribution_id,
+  UNIX_DATE(DATE(safe_parse_timestamp(first_run_date))) AS profile_creation_date,
+  region_home_region,
+  usage_is_default_browser,
+  search_engine_default_display_name,
+  search_engine_default_load_path,
+  search_engine_default_partner_code,
+  search_engine_default_provider_id,
+  search_engine_default_submission_url,
+  search_engine_default_overridden_by_third_party,
+  search_engine_private_display_name,
+  search_engine_private_load_path,
+  search_engine_private_partner_code,
+  search_engine_private_provider_id,
+  search_engine_private_submission_url,
+  search_engine_private_overridden_by_third_party,
+  CAST(overridden_by_third_party AS boolean) AS overridden_by_third_party,
+  subsession_start_time,
+  subsession_end_time,
+  subsession_counter,
+  experiments
+FROM
+  `moz-fx-data-shared-prod.firefox_desktop_derived.serp_events_v2`
+WHERE
+  submission_date
+  BETWEEN '2025-06-25'
+  AND '2025-09-25'
+  AND sample_id = 0
+-- submission_date = @submission_date
+QUALIFY
+  ROW_NUMBER() OVER (
+    PARTITION BY
+      client_id,
+      submission_date,
+      serp_provider_id,
+      serp_search_access_point
+    ORDER BY
+      event_timestamp DESC
+  ) = 1
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/09_serp_events_clients_ad_enterprise.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/09_serp_events_clients_ad_enterprise.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/09_serp_events_clients_ad_enterprise.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/09_serp_events_clients_ad_enterprise.sql	2025-11-25 01:10:52.000000000 +0000
@@ -0,0 +1,53 @@
+-- serp_events_clients_ad_enterprise_cte
+SELECT
+  serp_events_with_client_info_cte.client_id,
+  serp_events_with_client_info_cte.submission_date,
+  serp_events_with_client_info_cte.serp_provider_id,
+  serp_events_with_client_info_cte.partner_code,
+  serp_events_with_client_info_cte.serp_search_access_point,
+  serp_events_with_client_info_cte.sample_id,
+  serp_events_with_client_info_cte.profile_group_id,
+  serp_events_with_client_info_cte.legacy_telemetry_client_id,
+  serp_events_with_client_info_cte.country,
+  serp_events_with_client_info_cte.app_version,
+  serp_events_with_client_info_cte.channel,
+  serp_events_with_client_info_cte.normalized_channel,
+  serp_events_with_client_info_cte.locale,
+  serp_events_with_client_info_cte.os,
+  serp_events_with_client_info_cte.normalized_os,
+  serp_events_with_client_info_cte.os_version,
+  serp_events_with_client_info_cte.normalized_os_version,
+  serp_events_with_client_info_cte.os_version_major,
+  serp_events_with_client_info_cte.os_version_minor,
+  serp_events_with_client_info_cte.windows_build_number,
+  serp_events_with_client_info_cte.distribution_id,
+  serp_events_with_client_info_cte.profile_creation_date,
+  serp_events_with_client_info_cte.region_home_region,
+  serp_events_with_client_info_cte.usage_is_default_browser,
+  serp_events_with_client_info_cte.search_engine_default_display_name,
+  serp_events_with_client_info_cte.search_engine_default_load_path,
+  serp_events_with_client_info_cte.search_engine_default_partner_code,
+  serp_events_with_client_info_cte.search_engine_default_provider_id,
+  serp_events_with_client_info_cte.search_engine_default_submission_url,
+  serp_events_with_client_info_cte.search_engine_default_overridden_by_third_party,
+  serp_events_with_client_info_cte.search_engine_private_display_name,
+  serp_events_with_client_info_cte.search_engine_private_load_path,
+  serp_events_with_client_info_cte.search_engine_private_partner_code,
+  serp_events_with_client_info_cte.search_engine_private_provider_id,
+  serp_events_with_client_info_cte.search_engine_private_submission_url,
+  serp_events_with_client_info_cte.search_engine_private_overridden_by_third_party,
+  serp_events_with_client_info_cte.overridden_by_third_party,
+  serp_events_with_client_info_cte.subsession_start_time,
+  serp_events_with_client_info_cte.subsession_end_time,
+  serp_events_with_client_info_cte.subsession_counter,
+  serp_events_with_client_info_cte.experiments,
+  clients_with_adblocker_addons_cte.has_adblocker_addon,
+  serp_is_enterprise_cte.policies_is_enterprise
+FROM
+  `search_derived.search_clients_daily_v9.serp_events_with_client_info_cte`
+LEFT JOIN
+  `search_derived.search_clients_daily_v9.clients_with_adblocker_addons_cte`
+  USING (client_id, submission_date)
+LEFT JOIN
+  `search_derived.search_clients_daily_v9.serp_is_enterprise_cte`
+  USING (client_id, submission_date)
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/10_serp_ad_click_target.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/10_serp_ad_click_target.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/10_serp_ad_click_target.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/10_serp_ad_click_target.sql	2025-11-25 01:10:52.000000000 +0000
@@ -0,0 +1,24 @@
+-- serp_ad_click_target_cte
+SELECT
+  glean_client_id AS client_id,
+  submission_date,
+  `moz-fx-data-shared-prod.udf.normalize_search_engine`(
+    search_engine
+  ) AS serp_provider_id, -- this is engine
+  sap_source AS serp_search_access_point,
+  STRING_AGG(DISTINCT ad_components.component, ', ') AS ad_click_target
+FROM
+  `mozdata.firefox_desktop.serp_events`
+CROSS JOIN
+  UNNEST(ad_components) AS ad_components
+WHERE
+  submission_date
+  BETWEEN '2025-06-25'
+  AND '2025-09-25'
+  AND sample_id = 0
+-- submission_date = @submission_date
+GROUP BY
+  client_id,
+  submission_date,
+  serp_provider_id,
+  serp_search_access_point
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/11_serp_aggregates.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/11_serp_aggregates.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/11_serp_aggregates.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/11_serp_aggregates.sql	2025-11-25 01:10:52.000000000 +0000
@@ -0,0 +1,72 @@
+CREATE TEMP FUNCTION safe_parse_timestamp(ts string) AS (
+  COALESCE(
+        -- full datetime with offset
+    SAFE.PARSE_TIMESTAMP("%F%T%Ez", ts),
+        -- date + offset (no time)
+    SAFE.PARSE_TIMESTAMP("%F%Ez", ts),
+        -- datetime with space before offset
+    SAFE.PARSE_TIMESTAMP("%F%T%Ez", REGEXP_REPLACE(ts, r"(\+|\-)(\d{2}):(\d{2})", "\\1\\2\\3"))
+  )
+);
+
+CREATE TEMP FUNCTION to_utc_string(ts STRING) AS (
+  FORMAT_TIMESTAMP(
+    '%F %T UTC',  -- desired output format
+    SAFE.PARSE_TIMESTAMP('%FT%H:%M:%E*S%Ez', ts),
+    'UTC'
+  )
+);
+
+-- serp_aggregates_cte
+SELECT
+  glean_client_id AS client_id,
+  submission_date,
+  `moz-fx-data-shared-prod.udf.normalize_search_engine`(
+    search_engine
+  ) AS serp_provider_id, -- this is engine
+  sap_source AS serp_search_access_point,
+  LOGICAL_AND(ad_blocker_inferred) AS serp_ad_blocker_inferred,
+  COUNTIF(
+    (is_tagged IS TRUE)
+    AND (
+      sap_source = 'follow_on_from_refine_on_incontent_search'
+      OR sap_source = 'follow_on_from_refine_on_serp'
+    )
+  ) AS serp_follow_on_searches_tagged_count,
+  COUNTIF(is_tagged IS TRUE) AS serp_searches_tagged_count,
+  COUNTIF(is_tagged IS TRUE AND num_ads_visible > 0) AS serp_with_ads_tagged_count,
+  COUNTIF(is_tagged IS FALSE) AS serp_searches_organic_count,
+  COUNTIF(is_tagged IS FALSE AND num_ads_visible > 0) AS serp_with_ads_organic_count,
+  SUM(CASE WHEN is_tagged IS TRUE THEN num_ad_clicks ELSE 0 END) AS serp_ad_clicks_tagged_count,
+  SUM(CASE WHEN is_tagged IS FALSE THEN num_ad_clicks ELSE 0 END) AS serp_ad_clicks_organic_count,
+  SUM(num_ad_clicks) AS num_ad_clicks,
+  SUM(num_non_ad_link_clicks) AS num_non_ad_link_clicks,
+  SUM(num_other_engagements) AS num_other_engagements,
+  SUM(num_ads_loaded) AS num_ads_loaded,
+  SUM(num_ads_visible) AS num_ads_visible,
+  SUM(num_ads_blocked) AS num_ads_blocked,
+  SUM(num_ads_notshowing) AS num_ads_notshowing,
+  MAX(UNIX_DATE(DATE(to_utc_string(subsession_start_time)))) - MAX(
+    UNIX_DATE(DATE(safe_parse_timestamp(first_run_date)))
+  ) AS profile_age_in_days,
+  COUNT(*) AS serp_counts,
+  COUNTIF(subsession_counter = 1) AS sessions_started_on_this_day,
+  SUM(browser_engagement_active_ticks / (3600 / 5)) AS active_hours_sum,
+  SUM(
+    browser_engagement_tab_open_event_count
+  ) AS serp_scalar_parent_browser_engagement_tab_open_event_count_sum,
+  SUM(browser_engagement_uri_count) AS serp_scalar_parent_browser_engagement_total_uri_count_sum,
+  MAX(browser_engagement_max_concurrent_tab_count) AS max_concurrent_tab_count_max
+FROM
+  `mozdata.firefox_desktop.serp_events` -- serp_events_v2 doesn't have the aggregated fields like `num_ads_visible`
+WHERE
+  submission_date
+  BETWEEN '2025-06-25'
+  AND '2025-09-25'
+  AND sample_id = 0
+-- submission_date = @submission_date
+GROUP BY
+  client_id,
+  submission_date,
+  serp_provider_id,
+  serp_search_access_point
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/12_serp_final.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/12_serp_final.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/12_serp_final.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/12_serp_final.sql	2025-11-25 01:10:52.000000000 +0000
@@ -0,0 +1,76 @@
+-- final_serp_cte
+SELECT
+  serp_events_clients_ad_enterprise_cte.client_id,
+  serp_events_clients_ad_enterprise_cte.submission_date,
+  serp_events_clients_ad_enterprise_cte.serp_provider_id,
+  serp_events_clients_ad_enterprise_cte.partner_code,
+  serp_events_clients_ad_enterprise_cte.serp_search_access_point,
+  serp_events_clients_ad_enterprise_cte.sample_id,
+  serp_events_clients_ad_enterprise_cte.profile_group_id,
+  serp_events_clients_ad_enterprise_cte.legacy_telemetry_client_id,
+  serp_events_clients_ad_enterprise_cte.country,
+  serp_events_clients_ad_enterprise_cte.app_version,
+  serp_events_clients_ad_enterprise_cte.channel,
+  serp_events_clients_ad_enterprise_cte.normalized_channel,
+  serp_events_clients_ad_enterprise_cte.locale,
+  serp_events_clients_ad_enterprise_cte.os,
+  serp_events_clients_ad_enterprise_cte.normalized_os,
+  serp_events_clients_ad_enterprise_cte.os_version,
+  serp_events_clients_ad_enterprise_cte.normalized_os_version,
+  serp_events_clients_ad_enterprise_cte.os_version_major,
+  serp_events_clients_ad_enterprise_cte.os_version_minor,
+  serp_events_clients_ad_enterprise_cte.windows_build_number,
+  serp_events_clients_ad_enterprise_cte.distribution_id,
+  serp_events_clients_ad_enterprise_cte.profile_creation_date,
+  serp_events_clients_ad_enterprise_cte.region_home_region,
+  serp_events_clients_ad_enterprise_cte.usage_is_default_browser,
+  serp_events_clients_ad_enterprise_cte.search_engine_default_display_name,
+  serp_events_clients_ad_enterprise_cte.search_engine_default_load_path,
+  serp_events_clients_ad_enterprise_cte.search_engine_default_partner_code,
+  serp_events_clients_ad_enterprise_cte.search_engine_default_provider_id,
+  serp_events_clients_ad_enterprise_cte.search_engine_default_submission_url,
+  serp_events_clients_ad_enterprise_cte.search_engine_default_overridden_by_third_party,
+  serp_events_clients_ad_enterprise_cte.search_engine_private_display_name,
+  serp_events_clients_ad_enterprise_cte.search_engine_private_load_path,
+  serp_events_clients_ad_enterprise_cte.search_engine_private_partner_code,
+  serp_events_clients_ad_enterprise_cte.search_engine_private_provider_id,
+  serp_events_clients_ad_enterprise_cte.search_engine_private_submission_url,
+  serp_events_clients_ad_enterprise_cte.search_engine_private_overridden_by_third_party,
+  serp_events_clients_ad_enterprise_cte.overridden_by_third_party,
+  serp_events_clients_ad_enterprise_cte.subsession_start_time,
+  serp_events_clients_ad_enterprise_cte.subsession_end_time,
+  serp_events_clients_ad_enterprise_cte.subsession_counter,
+  serp_events_clients_ad_enterprise_cte.experiments,
+  serp_events_clients_ad_enterprise_cte.has_adblocker_addon,
+  serp_events_clients_ad_enterprise_cte.policies_is_enterprise,
+  serp_ad_click_target_cte.ad_click_target,
+  serp_aggregates_cte.serp_ad_blocker_inferred,
+  serp_aggregates_cte.serp_follow_on_searches_tagged_count,
+  serp_aggregates_cte.serp_searches_tagged_count,
+  serp_aggregates_cte.serp_searches_organic_count,
+  serp_aggregates_cte.serp_with_ads_organic_count,
+  serp_aggregates_cte.serp_with_ads_tagged_count,
+  serp_aggregates_cte.serp_ad_clicks_tagged_count,
+  serp_aggregates_cte.serp_ad_clicks_organic_count,
+  serp_aggregates_cte.num_ad_clicks,
+  serp_aggregates_cte.num_non_ad_link_clicks,
+  serp_aggregates_cte.num_other_engagements,
+  serp_aggregates_cte.num_ads_loaded,
+  serp_aggregates_cte.num_ads_visible,
+  serp_aggregates_cte.num_ads_blocked,
+  serp_aggregates_cte.num_ads_notshowing,
+  serp_aggregates_cte.profile_age_in_days,
+  serp_aggregates_cte.serp_counts,
+  serp_aggregates_cte.sessions_started_on_this_day,
+  serp_aggregates_cte.active_hours_sum,
+  serp_aggregates_cte.serp_scalar_parent_browser_engagement_tab_open_event_count_sum,
+  serp_aggregates_cte.serp_scalar_parent_browser_engagement_total_uri_count_sum,
+  serp_aggregates_cte.max_concurrent_tab_count_max
+FROM
+  `search_derived.search_clients_daily_v9.serp_events_clients_ad_enterprise_cte`
+LEFT JOIN
+  `search_derived.search_clients_daily_v9.serp_ad_click_target_cte`
+  USING (client_id, submission_date, serp_provider_id, serp_search_access_point)
+LEFT JOIN
+  `search_derived.search_clients_daily_v9.serp_aggregates_cte`
+  USING (client_id, submission_date, serp_provider_id, serp_search_access_point)
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/13_join_sap_serp.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/13_join_sap_serp.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/13_join_sap_serp.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/13_join_sap_serp.sql	2025-11-25 01:10:52.000000000 +0000
@@ -0,0 +1,150 @@
+-- join_sap_serp_cte
+SELECT
+  sap_final_cte.client_id AS sap_client_id,
+  serp_final_cte.client_id AS serp_client_id,
+  sap_final_cte.submission_date AS sap_submission_date,
+  serp_final_cte.submission_date AS serp_submission_date,
+  sap_final_cte.normalized_engine AS sap_normalized_engine,
+  serp_final_cte.serp_provider_id,
+  sap_final_cte.partner_code AS sap_partner_code,
+  serp_final_cte.partner_code AS serp_partner_code,
+  sap_final_cte.source AS sap_search_access_point,
+  serp_final_cte.serp_search_access_point,
+  sap_final_cte.sample_id AS sap_sample_id,
+  serp_final_cte.sample_id AS serp_sample_id,
+  sap_final_cte.profile_group_id AS sap_profile_group_id,
+  sap_final_cte.legacy_telemetry_client_id AS sap_legacy_telemetry_client_id,
+  serp_final_cte.legacy_telemetry_client_id AS serp_legacy_telemetry_client_id,
+  sap_final_cte.country AS sap_country,
+  sap_final_cte.app_version AS sap_app_version,
+  sap_final_cte.channel AS sap_channel,
+  sap_final_cte.normalized_channel AS sap_normalized_channel,
+  sap_final_cte.locale AS sap_locale,
+  sap_final_cte.os AS sap_os,
+  sap_final_cte.normalized_os AS sap_normalized_os,
+  sap_final_cte.os_version AS sap_os_version,
+  sap_final_cte.normalized_os_version AS sap_normalized_os_version,
+  sap_final_cte.windows_build_number AS sap_windows_build_number,
+  sap_final_cte.distribution_id AS sap_distribution_id,
+  sap_final_cte.profile_creation_date AS sap_profile_creation_date,
+  sap_final_cte.region_home_region AS sap_region_home_region,
+  sap_final_cte.usage_is_default_browser AS sap_usage_is_default_browser,
+  sap_final_cte.search_engine_default_display_name AS sap_default_search_engine_display_name,
+  sap_final_cte.search_engine_default_load_path AS sap_default_search_engine_load_path,
+  sap_final_cte.search_engine_default_partner_code AS sap_default_search_engine_partner_code,
+  sap_final_cte.search_engine_default_provider_id AS sap_default_search_engine_provider_id,
+  sap_final_cte.search_engine_default_submission_url AS sap_default_search_engine_submission_url,
+  sap_final_cte.search_engine_default_overridden_by_third_party AS sap_default_search_engine_overridden,
+  sap_final_cte.search_engine_private_display_name AS sap_default_private_search_engine_display_name,
+  sap_final_cte.search_engine_private_load_path AS sap_default_private_search_engine_load_path,
+  sap_final_cte.search_engine_private_partner_code AS sap_default_private_search_engine_partner_code,
+  sap_final_cte.search_engine_private_provider_id AS sap_default_private_search_engine_provider_id,
+  sap_final_cte.search_engine_private_submission_url AS sap_default_private_search_engine_submission_url,
+  sap_final_cte.search_engine_private_overridden_by_third_party AS sap_default_private_search_engine_overridden,
+  sap_final_cte.provider_name AS sap_provider_name,
+  sap_final_cte.provider_id AS sap_provider_id,
+  sap_final_cte.overridden_by_third_party AS sap_overridden_by_third_party,
+  sap_final_cte.subsession_start_time AS sap_subsession_start_time,
+  sap_final_cte.subsession_end_time AS sap_subsession_end_time,
+  sap_final_cte.subsession_counter AS sap_subsession_counter,
+  sap_final_cte.experiments AS sap_experiments,
+  sap_final_cte.has_adblocker_addon AS sap_has_adblocker_addon,
+  sap_final_cte.policies_is_enterprise AS sap_policies_is_enterprise,
+  sap_final_cte.profile_age_in_days AS sap_profile_age_in_days,
+  sap_final_cte.sap,
+  sap_final_cte.sessions_started_on_this_day AS sap_sessions_started_on_this_day,
+  sap_final_cte.active_hours_sum AS sap_active_hours_sum,
+  sap_final_cte.scalar_parent_browser_engagement_tab_open_event_count_sum AS sap_scalar_parent_browser_engagement_tab_open_event_count_sum,
+  sap_final_cte.scalar_parent_browser_engagement_total_uri_count_sum AS sap_scalar_parent_browser_engagement_total_uri_count_sum,
+  sap_final_cte.concurrent_tab_count_max AS sap_concurrent_tab_count_max,
+  serp_final_cte.profile_group_id AS serp_profile_group_id,
+  serp_final_cte.country AS serp_country,
+  serp_final_cte.app_version AS serp_app_version,
+  serp_final_cte.channel AS serp_channel,
+  serp_final_cte.normalized_channel AS serp_normalized_channel,
+  serp_final_cte.locale AS serp_locale,
+  serp_final_cte.os AS serp_os,
+  serp_final_cte.normalized_os AS serp_normalized_os,
+  serp_final_cte.os_version AS serp_os_version,
+  serp_final_cte.normalized_os_version AS serp_normalized_os_version,
+  serp_final_cte.os_version_major AS serp_os_version_major,
+  serp_final_cte.os_version_minor AS serp_os_version_minor,
+  serp_final_cte.windows_build_number AS serp_windows_build_number,
+  serp_final_cte.distribution_id AS serp_distribution_id,
+  serp_final_cte.profile_creation_date AS serp_profile_creation_date,
+  serp_final_cte.region_home_region AS serp_region_home_region,
+  serp_final_cte.usage_is_default_browser AS serp_usage_is_default_browser,
+  serp_final_cte.search_engine_default_display_name AS serp_default_search_engine_display_name,
+  serp_final_cte.search_engine_default_load_path AS serp_default_search_engine_load_path,
+  serp_final_cte.search_engine_default_partner_code AS serp_default_search_engine_partner_code,
+  serp_final_cte.search_engine_default_provider_id AS serp_default_search_engine_provider_id,
+  serp_final_cte.search_engine_default_submission_url AS serp_default_search_engine_submission_url,
+  serp_final_cte.search_engine_default_overridden_by_third_party AS serp_default_search_engine_overridden,
+  serp_final_cte.search_engine_private_display_name AS serp_default_private_search_engine_display_name,
+  serp_final_cte.search_engine_private_load_path AS serp_default_private_search_engine_load_path,
+  serp_final_cte.search_engine_private_partner_code AS serp_default_private_search_engine_partner_code,
+  serp_final_cte.search_engine_private_provider_id AS serp_default_private_search_engine_provider_id,
+  serp_final_cte.search_engine_private_submission_url AS serp_default_private_search_engine_submission_url,
+  serp_final_cte.search_engine_private_overridden_by_third_party AS serp_default_private_search_engine_overridden,
+  serp_final_cte.overridden_by_third_party AS serp_overridden_by_third_party,
+  serp_final_cte.subsession_start_time AS serp_subsession_start_time,
+  serp_final_cte.subsession_end_time AS serp_subsession_end_time,
+  serp_final_cte.subsession_counter AS serp_subsession_counter,
+  serp_final_cte.experiments AS serp_experiments,
+  serp_final_cte.has_adblocker_addon AS serp_has_adblocker_addon,
+  serp_final_cte.policies_is_enterprise AS serp_policies_is_enterprise,
+  serp_final_cte.ad_click_target AS serp_ad_click_target,
+  serp_final_cte.serp_ad_blocker_inferred AS serp_ad_blocker_inferred,
+  serp_final_cte.serp_follow_on_searches_tagged_count AS serp_follow_on_searches_tagged_count,
+  serp_final_cte.serp_searches_tagged_count AS serp_searches_tagged_count,
+  serp_final_cte.serp_searches_organic_count AS serp_searches_organic_count,
+  serp_final_cte.serp_with_ads_organic_count AS serp_with_ads_organic_count,
+  serp_final_cte.serp_with_ads_tagged_count AS serp_with_ads_tagged_count,
+  serp_final_cte.serp_ad_clicks_tagged_count AS serp_ad_clicks_tagged_count,
+  serp_final_cte.serp_ad_clicks_organic_count AS serp_ad_clicks_organic_count,
+  serp_final_cte.num_ad_clicks AS serp_num_ad_clicks,
+  serp_final_cte.num_non_ad_link_clicks AS serp_num_non_ad_link_clicks,
+  serp_final_cte.num_other_engagements AS serp_num_other_engagements,
+  serp_final_cte.num_ads_loaded AS serp_num_ads_loaded,
+  serp_final_cte.num_ads_visible AS serp_num_ads_visible,
+  serp_final_cte.num_ads_blocked AS serp_num_ads_blocked,
+  serp_final_cte.num_ads_notshowing AS serp_num_ads_notshowing,
+  serp_final_cte.profile_age_in_days AS serp_profile_age_in_days,
+  serp_final_cte.serp_counts,
+  serp_final_cte.sessions_started_on_this_day AS serp_sessions_started_on_this_day,
+  serp_final_cte.active_hours_sum AS serp_active_hours_sum,
+  serp_final_cte.serp_scalar_parent_browser_engagement_tab_open_event_count_sum AS serp_scalar_parent_browser_engagement_tab_open_event_count_sum,
+  serp_final_cte.serp_scalar_parent_browser_engagement_total_uri_count_sum AS serp_scalar_parent_browser_engagement_total_uri_count_sum,
+  serp_final_cte.max_concurrent_tab_count_max AS serp_max_concurrent_tab_count_max
+FROM
+  `search_derived.search_clients_daily_v9.sap_final_cte`
+FULL OUTER JOIN
+  `search_derived.search_clients_daily_v9.serp_final_cte`
+  ON (
+    sap_final_cte.client_id = serp_final_cte.client_id
+    OR (sap_final_cte.client_id IS NULL AND serp_final_cte.client_id IS NULL)
+  )
+  AND (
+    sap_final_cte.submission_date = serp_final_cte.submission_date
+    OR (sap_final_cte.submission_date IS NULL AND serp_final_cte.submission_date IS NULL)
+  )
+  AND (
+    sap_final_cte.normalized_engine = serp_final_cte.serp_provider_id
+    OR (sap_final_cte.normalized_engine IS NULL AND serp_final_cte.serp_provider_id IS NULL)
+  )
+  AND (
+    sap_final_cte.source = serp_final_cte.serp_search_access_point
+    OR (sap_final_cte.source IS NULL AND serp_final_cte.serp_search_access_point IS NULL)
+  )
+-- and sap_final_cte.search_access_point = serp_final_cte.serp_search_access_point -- rename)
+WHERE
+  sap_final_cte.normalized_channel IN ('nightly', 'beta')
+  OR serp_final_cte.normalized_channel IN ('nightly', 'beta')
+  AND client_id IN (
+    '28ea5671-012b-4207-af58-0af5af2fa1ae',
+    '7c3b8c4f-900b-424a-96f8-b3d0bbf7757c',
+    'f811c097-b567-419e-bc21-2f50783eb468',
+    'a90a56a1-0a82-41d0-8d9b-9cd906196a2f'
+  )
+ORDER BY
+  client_id
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/14_consolidated.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/14_consolidated.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/14_consolidated.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/14_consolidated.sql	2025-11-25 01:10:52.000000000 +0000
@@ -0,0 +1,98 @@
+SELECT
+  serp_submission_date AS submission_date,
+  serp_client_id AS client_id,
+  serp_legacy_telemetry_client_id AS legacy_telemetry_client_id, -- NEW
+  serp_provider_id AS engine, -- this is normalized in serp_events_with_client_info
+  CASE
+    WHEN serp_provider_id IS NULL
+      AND sap_normalized_engine IS NOT NULL
+      THEN sap_normalized_engine
+    ELSE serp_provider_id
+  END AS normalized_engine,
+  serp_search_access_point AS source,
+  serp_partner_code AS partner_code, -- NEW
+  serp_country AS country,
+  NULL AS addon_version,
+  serp_app_version AS app_version,
+  serp_windows_build_number AS windows_build_number, -- NEW
+  serp_distribution_id AS distribution_id,
+  serp_locale AS locale,
+  serp_region_home_region AS user_pref_browser_search_region,
+  NULL AS search_cohort,
+  serp_os AS os,
+  serp_normalized_os AS normalized_os, -- NEW
+  serp_os_version AS os_version,
+  serp_normalized_os_version AS normalized_os_version, -- NEW
+  serp_channel AS channel,
+  serp_normalized_channel AS normalized_channel, -- NEW
+  serp_usage_is_default_browser AS is_default_browser,
+  serp_profile_creation_date AS profile_creation_date,
+  serp_default_search_engine_display_name AS default_search_engine,
+  serp_default_search_engine_load_path AS default_search_engine_data_load_path,
+  serp_default_search_engine_submission_url AS default_search_engine_data_submission_url,
+  serp_default_search_engine_partner_code AS default_search_engine_partner_code, -- NEW
+  serp_default_search_engine_provider_id AS default_search_engine_provider_id, -- NEW
+  serp_default_search_engine_overridden AS default_search_engine_overridden, -- NEW
+  serp_default_private_search_engine_display_name AS default_private_search_engine,
+  serp_default_private_search_engine_load_path AS default_private_search_engine_data_load_path,
+  serp_default_private_search_engine_submission_url AS default_private_search_engine_data_submission_url,
+  serp_default_private_search_engine_partner_code AS default_private_search_engine_partner_code, -- NEW
+  serp_default_private_search_engine_provider_id AS default_private_search_engine_provider_id, -- NEW
+  serp_default_private_search_engine_overridden AS default_private_search_engine_overridden, -- NEW
+  serp_sample_id AS sample_id,
+  serp_subsession_start_time AS subsession_start_time, -- NEW
+  serp_subsession_end_time AS subsession_end_time, -- NEW
+  serp_subsession_counter AS subsession_counter, -- NEW
+  NULL AS subsessions_hours_sum,
+  serp_sessions_started_on_this_day AS sessions_started_on_this_day,
+  serp_overridden_by_third_party AS overridden_by_third_party, -- NEW
+  NULL AS active_addons_count_mean,
+  serp_max_concurrent_tab_count_max AS max_concurrent_tab_count_max,
+  serp_scalar_parent_browser_engagement_tab_open_event_count_sum AS tab_open_event_count_sum,
+  serp_active_hours_sum AS active_hours_sum,
+  serp_scalar_parent_browser_engagement_total_uri_count_sum AS total_uri_count,
+  serp_experiments AS experiments,
+  NULL AS scalar_parent_urlbar_searchmode_bookmarkmenu_sum,
+  NULL AS scalar_parent_urlbar_searchmode_handoff_sum,
+  NULL AS scalar_parent_urlbar_searchmode_keywordoffer_sum,
+  NULL AS scalar_parent_urlbar_searchmode_oneoff_sum,
+  NULL AS scalar_parent_urlbar_searchmode_other_sum,
+  NULL AS scalar_parent_urlbar_searchmode_shortcut_sum,
+  NULL AS scalar_parent_urlbar_searchmode_tabmenu_sum,
+  NULL AS scalar_parent_urlbar_searchmode_tabtosearch_sum,
+  NULL AS scalar_parent_urlbar_searchmode_tabtosearch_onboard_sum,
+  NULL AS scalar_parent_urlbar_searchmode_topsites_newtab_sum,
+  NULL AS scalar_parent_urlbar_searchmode_topsites_urlbar_sum,
+  NULL AS scalar_parent_urlbar_searchmode_touchbar_sum,
+  NULL AS scalar_parent_urlbar_searchmode_typed_sum,
+  serp_profile_age_in_days AS profile_age_in_days,
+  serp_searches_organic_count AS organic,
+  serp_searches_tagged_count AS tagged_sap,
+  serp_searches_tagged_count AS tagged_serp,
+  serp_follow_on_searches_tagged_count AS tagged_follow_on,
+  sap,
+  serp_counts,
+  serp_ad_click_target AS ad_click_target,
+  serp_num_ad_clicks AS ad_click,
+  serp_ad_clicks_organic_count AS ad_click_organic,
+  serp_with_ads_tagged_count AS search_with_ads,
+  serp_with_ads_organic_count AS search_with_ads_organic,
+  serp_ad_clicks_tagged_count AS ad_clicks_tagged,
+  serp_ad_blocker_inferred AS ad_blocker_inferred,
+  serp_num_non_ad_link_clicks AS num_non_ad_link_clicks, -- NEW
+  serp_num_other_engagements AS num_other_engagements, -- NEW
+  serp_num_ads_loaded AS num_ads_loaded, -- NEW
+  serp_num_ads_visible AS num_ads_visible, -- NEW
+  serp_num_ads_blocked AS num_ads_blocked, -- NEW
+  serp_num_ads_notshowing AS num_ads_notshowing, -- NEW
+  NULL AS unknown,
+  NULL AS is_sap_monetizable, -- REVISIT
+  serp_has_adblocker_addon AS has_adblocker_addon,
+  serp_policies_is_enterprise AS policies_is_enterprise,
+  serp_os_version_major AS os_version_major,
+  serp_os_version_minor AS os_version_minor,
+  serp_profile_group_id AS profile_group_id
+FROM
+  `search_derived.search_clients_daily_v9.join_sap_serp_cte`
+WHERE
+  serp_submission_date IS NOT NULL
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/ctes/metadata.yaml	2025-11-25 01:12:08.000000000 +0000
@@ -0,0 +1,40 @@
+
+# Generated by bigquery_etl.dependency
+references:
+  01_adblocker.sql:
+  - moz-fx-data-shared-prod.firefox_desktop_stable.metrics_v1
+  - moz-fx-data-shared-prod.revenue.monetization_blocking_addons
+  02_sap_is_enterprise.sql:
+  - moz-fx-data-shared-prod.firefox_desktop_derived.events_stream_v1
+  03_sap_events_with_client_info.sql:
+  - moz-fx-data-shared-prod.firefox_desktop_derived.events_stream_v1
+  04_sap_events_clients_ad_enterprise.sql:
+  - search_derived.search_clients_daily_v9.clients_with_adblocker_addons_cte
+  - search_derived.search_clients_daily_v9.sap_events_with_client_info_cte
+  - search_derived.search_clients_daily_v9.sap_is_enterprise_cte
+  05_sap_aggregates.sql:
+  - moz-fx-data-shared-prod.firefox_desktop_derived.events_stream_v1
+  06_sap_final.sql:
+  - search_derived.search_clients_daily_v9.sap_aggregates_cte
+  - search_derived.search_clients_daily_v9.sap_events_clients_ad_enterprise_cte
+  07_serp_is_enterprise.sql:
+  - moz-fx-data-shared-prod.firefox_desktop_derived.serp_events_v2
+  08_serp_events_with_client_info.sql:
+  - moz-fx-data-shared-prod.firefox_desktop_derived.serp_events_v2
+  09_serp_events_clients_ad_enterprise.sql:
+  - search_derived.search_clients_daily_v9.clients_with_adblocker_addons_cte
+  - search_derived.search_clients_daily_v9.serp_events_with_client_info_cte
+  - search_derived.search_clients_daily_v9.serp_is_enterprise_cte
+  10_serp_ad_click_target.sql:
+  - mozdata.firefox_desktop.serp_events
+  11_serp_aggregates.sql:
+  - mozdata.firefox_desktop.serp_events
+  12_serp_final.sql:
+  - search_derived.search_clients_daily_v9.serp_ad_click_target_cte
+  - search_derived.search_clients_daily_v9.serp_aggregates_cte
+  - search_derived.search_clients_daily_v9.serp_events_clients_ad_enterprise_cte
+  13_join_sap_serp.sql:
+  - search_derived.search_clients_daily_v9.sap_final_cte
+  - search_derived.search_clients_daily_v9.serp_final_cte
+  14_consolidated.sql:
+  - search_derived.search_clients_daily_v9.join_sap_serp_cte
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/search_derived/search_clients_daily_v9/metadata.yaml	2025-11-25 01:12:15.000000000 +0000
@@ -0,0 +1,45 @@
+friendly_name: Search Clients Daily
+description: |
+  A daily aggregate of desktop searches per (client_id, engine, source).
+
+  Exposed to users as view `search.search_clients_engines_sources_daily`.
+owners:
+- akommasani@mozilla.com
+- kbammarito@mozilla.com
+labels:
+  schedule: daily
+  table_type: client_level
+  dag: bqetl_search
+  owner1: akommasani
+  owner2: kbammarito
+scheduling:
+  dag_name: bqetl_search
+  external_downstream_tasks:
+  - task_id: wait_for_search_clients_daily
+    dag_name: jetstream
+    execution_delta: 1h
+  - task_id: wait_for_search_clients_daily
+    dag_name: operational_monitoring
+    execution_delta: 1h
+bigquery:
+  time_partitioning:
+    type: day
+    field: submission_date
+    require_partition_filter: true
+    expiration_days: 775.0
+  range_partitioning: null
+  clustering:
+    fields:
+    - sample_id
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+references:
+  query.sql:
+  - moz-fx-

⚠️ Only part of the diff is displayed.

Link to full diff