Skip to content

Update prom scrape config to match dcgm-exporter pattern #228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sandipanpanda merged 1 commit into from
Jan 9, 2026
+13 −8
Merged

Update prom scrape config to match dcgm-exporter pattern #228

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions config/prometheus/hack.prometheus.values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -410,9 +410,9 @@ serverFiles:
regex: true
# ONLY scrape DCGM exporter pods to reduce load on Prometheus
# This prevents scraping random pods that have prometheus.io/scrape=true
- source_labels: [ __meta_kubernetes_pod_label_app ]
action: keep
regex: nvidia-dcgm-exporter
- action: keep
regex: '.*dcgm-exporter.*'
source_labels: [ __meta_kubernetes_pod_label_app, __meta_kubernetes_pod_label_app_kubernetes_io_name ]
- source_labels: [ __meta_kubernetes_pod_annotation_prometheus_io_scrape_slow ]
action: drop
regex: true
Expand Down
3 changes: 2 additions & 1 deletion dist/backend-install.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,9 +141,10 @@ data:
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
- action: keep
regex: nvidia-dcgm-exporter
regex: .*dcgm-exporter.*
source_labels:
- __meta_kubernetes_pod_label_app
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- action: drop
regex: true
source_labels:
Expand Down
3 changes: 2 additions & 1 deletion dist/install.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,9 +142,10 @@ data:
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
- action: keep
regex: nvidia-dcgm-exporter
regex: .*dcgm-exporter.*
source_labels:
- __meta_kubernetes_pod_label_app
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- action: drop
regex: true
source_labels:
Expand Down
3 changes: 2 additions & 1 deletion dist/installer_updater.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,9 +141,10 @@ data:
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
- action: keep
regex: nvidia-dcgm-exporter
regex: .*dcgm-exporter.*
source_labels:
- __meta_kubernetes_pod_label_app
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- action: drop
regex: true
source_labels:
Expand Down
3 changes: 2 additions & 1 deletion dist/prometheus.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,9 +126,10 @@ data:
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
- action: keep
regex: nvidia-dcgm-exporter
regex: .*dcgm-exporter.*
source_labels:
- __meta_kubernetes_pod_label_app
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- action: drop
regex: true
source_labels:
Expand Down
3 changes: 2 additions & 1 deletion helm-chart/zxporter/templates/prometheus-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,9 +90,10 @@ data:
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
- action: keep
regex: nvidia-dcgm-exporter
regex: '.*dcgm-exporter.*'
source_labels:
- __meta_kubernetes_pod_label_app
- __meta_kubernetes_pod_label_app_kubernetes_io_name
Comment on lines 92 to +96
Copy link

@coderabbitai coderabbitai bot Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

# Find and examine the prometheus-configmap.yaml file
find . -name "prometheus-configmap.yaml" -type f | head -20

Repository: devzero-inc/zxporter

Length of output: 121

🏁 Script executed:

# Once we find it, check the lines in question
git ls-files | grep -i prometheus-configmap

Repository: devzero-inc/zxporter

Length of output: 119

🏁 Script executed:

# Search for the file
fd prometheus-configmap.yaml

Repository: devzero-inc/zxporter

Length of output: 119

🏁 Script executed:

# Read the prometheus-configmap.yaml file, focusing on lines 92-96
cat -n ./helm-chart/zxporter/templates/prometheus-configmap.yaml | sed -n '80,110p'

Repository: devzero-inc/zxporter

Length of output: 1264

🏁 Script executed:

# Also check the git history to see what changed
git log --oneline -20 -- ./helm-chart/zxporter/templates/prometheus-configmap.yaml

Repository: devzero-inc/zxporter

Length of output: 164

🏁 Script executed:

# Check if there's a git diff to see the actual changes
git diff HEAD~1 ./helm-chart/zxporter/templates/prometheus-configmap.yaml | head -100

Repository: devzero-inc/zxporter

Length of output: 250

🏁 Script executed:

# Get the full commit to see the changes
git show 81bf4dc --stat

Repository: devzero-inc/zxporter

Length of output: 17165

🏁 Script executed:

# Get the actual diff of this commit
git show 81bf4dc

Repository: devzero-inc/zxporter

Length of output: 50378

🏁 Script executed:

# Check if there's any documentation or comments in the file about this
cat -n ./helm-chart/zxporter/templates/prometheus-configmap.yaml | head -100

Repository: devzero-inc/zxporter

Length of output: 4085

🏁 Script executed:

# Let me check the git commit to see what the previous state was
git show 81bf4dc:helm-chart/zxporter/templates/prometheus-configmap.yaml | sed -n '80,110p'

Repository: devzero-inc/zxporter

Length of output: 1047

🌐 Web query:

Prometheus relabel_config multiple source_labels concatenation separator

💡 Result:

Yes — when you list multiple source_labels in a relabel_config, Prometheus concatenates their values using the separator (defaults to ";"), then matches that combined string with your regex and uses capture groups in replacement. Example:

  • separator default: ";" (you can set any string).
  • flow: concatenate source_labels with separator → match regex → use $1, $2… in replacement.

Example:
relabel_configs:

  • source_labels: [job, instance]
    separator: "_"
    regex: (.+)
    replacement: "$1"
    target_label: combined

This will set label combined to "job_instance" (e.g. "api_10.0.0.1"). (prometheus.io)

Citations:

🌐 Web query:

Prometheus relabel action keep regex multiple source_labels behavior

💡 Result:

Short answer: When keep (or drop/replace/hashmod) has multiple source_labels, Prometheus concatenates their values with the configured separator (default ";"), then matches your regex against that single concatenated string (the regex is anchored on both ends). If the regex matches the concatenation the object is kept (keep) or dropped (drop); otherwise it is removed/kept accordingly. Example: source_labels: [app,region], separator: ":" and regex: "frontend:us-west" will match only when app=="frontend" and region=="us-west". [1][2]

Sources:

  • Prometheus docs — relabeling & config (concatenation, separator, anchored regex, actions). [1]
  • PromLabs relabeling guide (keep/drop behavior and examples). [2]

🏁 Script executed:

# Verify if there's a previous version of this file in git history
git log --all --full-history -- helm-chart/zxporter/templates/prometheus-configmap.yaml | head -20

Repository: devzero-inc/zxporter

Length of output: 757

🏁 Script executed:

# Check if the file appears in any previous commits
git log --follow --all -p -- helm-chart/zxporter/templates/prometheus-configmap.yaml | head -100

Repository: devzero-inc/zxporter

Length of output: 3546

🏁 Script executed:

# Check commit 65f1bcf to see if it had any prometheus config
git show 65f1bcf --stat | head -30

Repository: devzero-inc/zxporter

Length of output: 1022

🏁 Script executed:

# Search for any previous prometheus config file in the repo history
git log --all --full-history --diff-filter=D -- "*prometheus*configmap*" | head -20

Repository: devzero-inc/zxporter

Length of output: 46

🏁 Script executed:

# Check if there are other prometheus config files in the repo
find . -name "*prometheus*" -type f | grep -v ".git"

Repository: devzero-inc/zxporter

Length of output: 426

🏁 Script executed:

# Let's see what changed in the prometheus-configmap.yaml at commit 65f1bcf
git show 65f1bcf:helm-chart/zxporter/templates/prometheus-configmap.yaml | grep -A 20 "kubernetes-pods"

Repository: devzero-inc/zxporter

Length of output: 716

🏁 Script executed:

# And compare to commit 81bf4dc
git show 81bf4dc:helm-chart/zxporter/templates/prometheus-configmap.yaml | grep -A 20 "kubernetes-pods"

Repository: devzero-inc/zxporter

Length of output: 743

🏁 Script executed:

# Get the actual diff between these two commits for the configmap
git diff 65f1bcf 81bf4dc -- helm-chart/zxporter/templates/prometheus-configmap.yaml | head -100

Repository: devzero-inc/zxporter

Length of output: 749

The regex pattern change correctly supports GKE dcgm-exporter with flexible labeling.

The changes appropriately:

  1. Broaden the regex from literal nvidia-dcgm-exporter to '.*dcgm-exporter.*' to accommodate label variations across different Kubernetes distributions
  2. Add __meta_kubernetes_pod_label_app_kubernetes_io_name to handle GKE's standard labeling scheme

Prometheus concatenates multiple source_labels with a separator (default ;) and matches the regex against the combined string. This effectively creates an OR condition—the pod is scraped if either the app or app.kubernetes.io/name label contains "dcgm-exporter".

The broader regex pattern could match unintended pods (e.g., "test-dcgm-exporter-backup", "dcgm-exporter-v2-dev"), but the configuration remains protected by the prometheus.io/scrape=true annotation requirement (checked in the preceding relabel rule) and any downstream metric filtering.

🤖 Prompt for AI Agents 
In @helm-chart/zxporter/templates/prometheus-configmap.yaml around lines 92 -
96, The current relabeling broadened the regex to '.*dcgm-exporter.*' and added
the source label __meta_kubernetes_pod_label_app_kubernetes_io_name which can
match unintended pods; tighten the match by adjusting the relabel regex to only
match token boundaries or exact label values for dcgm-exporter (for example use
anchors or word-boundary-like patterns) so that combined source_labels (
__meta_kubernetes_pod_label_app and
__meta_kubernetes_pod_label_app_kubernetes_io_name ) only match intended
dcgm-exporter pods while keeping the existing prerequisite check for
prometheus.io/scrape=true.

- action: drop
regex: true
source_labels:
Expand Down
Loading