[MVP] feat(Autodiscovery): CRD-driven check scheduling via DatadogInstrumentation

[Mathew-Estafanous]

What does this PR do?

Adds end-to-end CRD-driven check scheduling: the Cluster Agent watches DatadogInstrumentation CRs, converts their spec.config.checks into autodiscovery-compatible YAML, writes the result to a ConfigMap, and the Node Agent picks it up via a new file-based config provider.

Two independent pieces work together:

• Cluster Agent — InstrumentationController (pkg/clusteragent/instrumentation/): A shared informer watches datadoginstrumentations CRDs. On each reconciliation cycle it converts all CRs, then fans out to registered ConfigSectionHandler implementations. Today the only handler is checks.Handler, which builds AD configs (with CEL selectors derived from spec.selector) and writes them to a single ConfigMap. The controller is leader-aware and idempotent.

• Node Agent — CRDFileConfigProvider (comp/core/autodiscovery/providers/crd_file.go): Reads YAML files from a directory (autoconf_crd_checks_dir, default /etc/datadog-agent/crd-conf.d) that is expected to be a ConfigMap volume mount. Files follow the naming convention <namespace>_<crname>_<checkname>.yaml. The provider is registered when podcheck.enabled is true.

The controller architecture is deliberately extensible. Adding a new config section (e.g., APM instrumentation) requires only implementing ConfigSectionHandler and adding it to the handler slice — no changes to the shared controller.

New config keys

Key	Default	Description
workload_config.enabled	true	Enables the CRD controller (Cluster Agent) and the CRD file provider (Node Agent)
autoconf_crd_checks_dir	/etc/datadog-agent/crd-conf.d	Directory the Node Agent reads for CRD-driven configs
autoconf_crd_checks_poll_interval	10 (seconds)	Poll interval for the CRD file provider

Motivation

Today, scheduling an AD check on a set of pods requires either Kubernetes annotations on the workload or static config files baked into the Agent image. Both approaches couple check configuration to the workload owner and don't scale well when a platform team wants to roll out checks across many workloads centrally.

DatadogInstrumentation (defined in datadog-operator PR #2724) lets a platform team declare checks as Kubernetes CRs, scoped to pods via label/annotation selectors. The Cluster Agent reconciles these CRs into a ConfigMap that is mounted into the Node Agent, closing the loop without any changes to workloads.

The generalized controller design prepares the path for future CRD-driven features to plug in alongside checks.

Describe how you validated your changes

• Unit tests for all new packages (go test -tags kubeapiserver ./pkg/clusteragent/instrumentation/...):
  • Controller: single CR, multiple CRs, empty list, non-leader skip, multiple handlers, handler error propagation
  • CEL selector generation: labels, annotations, combined selectors, sorted keys, empty maps
  • Check conversion: single/multiple checks, nil init_config, logs, invalid JSON
• Unit tests for the CRD file provider (comp/core/autodiscovery/providers/crd_file_test.go):
  • Name extraction, directory reading, non-YAML filtering, empty/missing directory handling

**TODO: manual validation against a local cluster

Additional Notes

• The DatadogInstrumentation CRD type comes from datadog-operator PR #2724. The operator dependency is pinned to the head of that PR branch (56ca338). This will need updating to a release tag before merge.
• The workloadfilter.CreatePod helper was extended to accept labels (previously only annotations), since CEL selectors now match on pod labels.

[agent-platform-auto-pr]

Go Package Import Differences

Baseline: 3002045
Comparison: 0a7b81e

binary	os	arch	change
cluster-agent	linux	amd64	+2, -0 +github.com/DataDog/datadog-agent/pkg/clusteragent/instrumentation +github.com/DataDog/datadog-agent/pkg/clusteragent/instrumentation/checks
cluster-agent	linux	arm64	+2, -0 +github.com/DataDog/datadog-agent/pkg/clusteragent/instrumentation +github.com/DataDog/datadog-agent/pkg/clusteragent/instrumentation/checks

[agent-platform-auto-pr]

Files inventory check summary

File checks results against ancestor 30020450:

Results for datadog-agent_7.78.0~devel.git.548.0a7b81e.pipeline.102404105-1_amd64.deb:

No change detected

[agent-platform-auto-pr]

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor 3002045
📊 Static Quality Gates Dashboard
🔗 SQG Job

Successful checks

Info

	Quality gate	Change	Size (prev → curr → max)
✅	agent_deb_amd64	+15.65 KiB (0.00% increase)	746.881 → 746.896 → 750.720
✅	agent_deb_amd64_fips	+15.65 KiB (0.00% increase)	705.213 → 705.228 → 711.230
✅	agent_heroku_amd64	+16.49 KiB (0.01% increase)	311.746 → 311.762 → 320.580
✅	agent_msi	+18.5 KiB (0.00% increase)	609.827 → 609.845 → 1062.520
✅	agent_rpm_amd64	+15.65 KiB (0.00% increase)	746.864 → 746.880 → 750.690
✅	agent_rpm_amd64_fips	+15.65 KiB (0.00% increase)	705.197 → 705.212 → 711.210
✅	agent_rpm_arm64	+9.06 KiB (0.00% increase)	724.745 → 724.754 → 732.890
✅	agent_rpm_arm64_fips	+21.06 KiB (0.00% increase)	686.083 → 686.103 → 694.440
✅	agent_suse_amd64	+15.65 KiB (0.00% increase)	746.864 → 746.880 → 750.690
✅	agent_suse_amd64_fips	+15.65 KiB (0.00% increase)	705.197 → 705.212 → 711.210
✅	agent_suse_arm64	+9.06 KiB (0.00% increase)	724.745 → 724.754 → 732.890
✅	agent_suse_arm64_fips	+21.06 KiB (0.00% increase)	686.083 → 686.103 → 694.440
✅	docker_agent_amd64	+26.79 KiB (0.00% increase)	807.202 → 807.228 → 813.040
✅	docker_agent_arm64	+20.19 KiB (0.00% increase)	809.888 → 809.908 → 819.570
✅	docker_agent_jmx_amd64	+26.79 KiB (0.00% increase)	998.118 → 998.144 → 1003.920
✅	docker_agent_jmx_arm64	+20.2 KiB (0.00% increase)	989.583 → 989.602 → 999.170
✅	docker_cluster_agent_amd64	+149.44 KiB (0.07% increase)	203.673 → 203.819 → 206.270
✅	docker_cluster_agent_arm64	+129.25 KiB (0.06% increase)	218.101 → 218.227 → 220.000
✅	docker_dogstatsd_amd64	+4.01 KiB (0.01% increase)	38.635 → 38.639 → 39.380
✅	dogstatsd_deb_arm64	+4.0 KiB (0.01% increase)	28.008 → 28.012 → 29.110
✅	iot_agent_deb_amd64	+12.0 KiB (0.03% increase)	43.072 → 43.084 → 43.290
✅	iot_agent_deb_arm64	+12.0 KiB (0.03% increase)	40.130 → 40.142 → 40.920
✅	iot_agent_deb_armhf	+8.02 KiB (0.02% increase)	40.874 → 40.882 → 41.030
✅	iot_agent_rpm_amd64	+12.0 KiB (0.03% increase)	43.072 → 43.084 → 43.290
✅	iot_agent_suse_amd64	+12.0 KiB (0.03% increase)	43.072 → 43.084 → 43.290

6 successful checks with minimal change (< 2 KiB)

	Quality gate	Current Size
✅	docker_cws_instrumentation_amd64	7.138 MiB
✅	docker_cws_instrumentation_arm64	6.689 MiB
✅	docker_dogstatsd_arm64	36.939 MiB
✅	dogstatsd_deb_amd64	29.859 MiB
✅	dogstatsd_rpm_amd64	29.859 MiB
✅	dogstatsd_suse_amd64	29.859 MiB

On-wire sizes (compressed)

	Quality gate	Change	Size (prev → curr → max)
✅	agent_deb_amd64	-9.85 KiB (0.01% reduction)	174.432 → 174.423 → 177.700
✅	agent_deb_amd64_fips	+3.82 KiB (0.00% increase)	165.351 → 165.355 → 172.230
✅	agent_heroku_amd64	-8.18 KiB (0.01% reduction)	75.180 → 75.172 → 79.970
✅	agent_msi	+24.0 KiB (0.02% increase)	138.008 → 138.031 → 146.220
✅	agent_rpm_amd64	-2.52 KiB (0.00% reduction)	176.433 → 176.430 → 180.780
✅	agent_rpm_amd64_fips	-60.64 KiB (0.04% reduction)	168.129 → 168.070 → 173.370
✅	agent_rpm_arm64	-6.25 KiB (0.00% reduction)	159.281 → 159.275 → 161.610
✅	agent_rpm_arm64_fips	-12.03 KiB (0.01% reduction)	151.447 → 151.435 → 155.910
✅	agent_suse_amd64	-2.52 KiB (0.00% reduction)	176.433 → 176.430 → 180.780
✅	agent_suse_amd64_fips	-60.64 KiB (0.04% reduction)	168.129 → 168.070 → 173.370
✅	agent_suse_arm64	-6.25 KiB (0.00% reduction)	159.281 → 159.275 → 161.610
✅	agent_suse_arm64_fips	-12.03 KiB (0.01% reduction)	151.447 → 151.435 → 155.910
✅	docker_agent_amd64	-11.98 KiB (0.00% reduction)	267.123 → 267.112 → 271.240
✅	docker_agent_arm64	+4.93 KiB (0.00% increase)	254.426 → 254.431 → 259.800
✅	docker_agent_jmx_amd64	-13.07 KiB (0.00% reduction)	335.768 → 335.755 → 339.870
✅	docker_agent_jmx_arm64	-16.72 KiB (0.01% reduction)	319.076 → 319.060 → 324.390
✅	docker_cluster_agent_amd64	+42.68 KiB (0.06% increase)	71.286 → 71.328 → 72.920
✅	docker_cluster_agent_arm64	+40.43 KiB (0.06% increase)	66.924 → 66.963 → 68.220
✅	docker_cws_instrumentation_amd64	neutral	2.997 MiB → 3.330
✅	docker_cws_instrumentation_arm64	neutral	2.728 MiB → 3.090
✅	docker_dogstatsd_amd64	neutral	14.950 MiB → 15.820
✅	docker_dogstatsd_arm64	+7.71 KiB (0.05% increase)	14.280 → 14.288 → 14.830
✅	dogstatsd_deb_amd64	neutral	7.890 MiB → 8.790
✅	dogstatsd_deb_arm64	neutral	6.774 MiB → 7.710
✅	dogstatsd_rpm_amd64	neutral	7.899 MiB → 8.800
✅	dogstatsd_suse_amd64	neutral	7.899 MiB → 8.800
✅	iot_agent_deb_amd64	neutral	11.355 MiB → 12.040
✅	iot_agent_deb_arm64	+2.02 KiB (0.02% increase)	9.670 → 9.672 → 10.450
✅	iot_agent_deb_armhf	+2.29 KiB (0.02% increase)	9.899 → 9.901 → 10.620
✅	iot_agent_rpm_amd64	neutral	11.375 MiB → 12.060
✅	iot_agent_suse_amd64	neutral	11.375 MiB → 12.060

[cit-pr-commenter-54b7da]

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: 47133925-14d9-400f-bdd1-bf56ec3b2a89

Baseline: 3002045
Comparison: 0a7b81e
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	+1.22	[-1.78, +4.22]	1	Logs

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	+1.22	[-1.78, +4.22]	1	Logs
➖	quality_gate_metrics_logs	memory utilization	+0.71	[+0.47, +0.94]	1	Logs bounds checks dashboard
➖	otlp_ingest_logs	memory utilization	+0.52	[+0.43, +0.61]	1	Logs
➖	ddot_metrics_sum_cumulativetodelta_exporter	memory utilization	+0.45	[+0.23, +0.68]	1	Logs
➖	uds_dogstatsd_20mb_12k_contexts_20_senders	memory utilization	+0.30	[+0.24, +0.36]	1	Logs
➖	otlp_ingest_metrics	memory utilization	+0.22	[+0.06, +0.39]	1	Logs
➖	quality_gate_idle	memory utilization	+0.10	[+0.05, +0.15]	1	Logs bounds checks dashboard
➖	ddot_metrics_sum_cumulative	memory utilization	+0.06	[-0.08, +0.20]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	+0.02	[-0.08, +0.12]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	+0.01	[-0.42, +0.45]	1	Logs
➖	uds_dogstatsd_to_api_v3	ingress throughput	+0.01	[-0.16, +0.19]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.01	[-0.09, +0.11]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	+0.00	[-0.49, +0.50]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	-0.01	[-0.20, +0.18]	1	Logs
➖	file_to_blackhole_500ms_latency	egress throughput	-0.05	[-0.43, +0.33]	1	Logs
➖	quality_gate_logs	% cpu utilization	-0.10	[-1.73, +1.53]	1	Logs bounds checks dashboard
➖	ddot_metrics	memory utilization	-0.14	[-0.32, +0.03]	1	Logs
➖	ddot_logs	memory utilization	-0.16	[-0.21, -0.10]	1	Logs
➖	docker_containers_memory	memory utilization	-0.18	[-0.26, -0.10]	1	Logs
➖	ddot_metrics_sum_delta	memory utilization	-0.22	[-0.39, -0.06]	1	Logs
➖	file_tree	memory utilization	-0.33	[-0.38, -0.27]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	-0.40	[-0.44, -0.36]	1	Logs bounds checks dashboard
➖	tcp_syslog_to_blackhole	ingress throughput	-2.06	[-2.19, -1.93]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	observed_value	links
✅	docker_containers_cpu	simple_check_run	10/10	708 ≥ 26
✅	docker_containers_memory	memory_usage	10/10	273.56MiB ≤ 370MiB
✅	docker_containers_memory	simple_check_run	10/10	596 ≥ 26
✅	file_to_blackhole_0ms_latency	memory_usage	10/10	0.19GiB ≤ 1.20GiB
✅	file_to_blackhole_0ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10	0.23GiB ≤ 1.20GiB
✅	file_to_blackhole_1000ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_100ms_latency	memory_usage	10/10	0.20GiB ≤ 1.20GiB
✅	file_to_blackhole_100ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_500ms_latency	memory_usage	10/10	0.21GiB ≤ 1.20GiB
✅	file_to_blackhole_500ms_latency	missed_bytes	10/10	0B = 0B
✅	quality_gate_idle	intake_connections	10/10	3 = 3	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	173.92MiB ≤ 175MiB	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	3 = 3	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	487.21MiB ≤ 550MiB	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	4 ≤ 6	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	203.56MiB ≤ 220MiB	bounds checks dashboard
✅	quality_gate_logs	missed_bytes	10/10	0B = 0B	bounds checks dashboard
✅	quality_gate_metrics_logs	cpu_usage	10/10	346.59 ≤ 2000	bounds checks dashboard
✅	quality_gate_metrics_logs	intake_connections	10/10	4 ≤ 6	bounds checks dashboard
✅	quality_gate_metrics_logs	memory_usage	10/10	409.08MiB ≤ 475MiB	bounds checks dashboard
✅	quality_gate_metrics_logs	missed_bytes	10/10	0B = 0B	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

• quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
• quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
• quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
• quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
• quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.