Skip to content

exporter: parallelize metrics generation and refresh in background#83

Open
qingqiuhe wants to merge 1 commit intoProject-HAMi:mainfrom
qingqiuhe:codex/exporter-parallel-cache
Open

exporter: parallelize metrics generation and refresh in background#83
qingqiuhe wants to merge 1 commit intoProject-HAMi:mainfrom
qingqiuhe:codex/exporter-parallel-cache

Conversation

@qingqiuhe
Copy link
Copy Markdown

Summary

This PR brings over the two main optimizations that exist in our fork but are still missing from GitHub main:

  1. Parallelize exporter metric collection loops with bounded concurrency.
  2. Replace on-demand /metrics generation with background periodic refresh.

What changed

  • parallelized device-level and container-level metrics collection in MetricsGenerator with errgroup.SetLimit(...)
  • added exporter_concurrency_limit config with default 16
  • moved metrics generation off the /metrics request path and into a background ticker
  • added metrics_generate_interval config with default 15s
  • exposed both settings in server/config/config.yaml and the Helm chart values/configmap
  • promoted golang.org/x/sync to a direct dependency in server/go.mod

Why

On the current GitHub main branch:

  • exporter collection is still largely serial inside the device/container loops
  • /metrics still triggers synchronous GenerateMetrics(...) on every scrape
  • the new concurrency and refresh interval settings are not wired through config/Helm

This means scrape latency can grow with cluster size, and Prometheus requests stay coupled to live collection latency.

Validation

  • cd server && go test ./internal/exporter ./internal/server ./cmd/server

Note: cd server && go test ./... still fails on an existing unrelated issue in internal/provider/util/util.go:332 (fmt.Sprintf argument mismatch), which is outside the scope of this PR.

@hami-robot
Copy link
Copy Markdown

hami-robot bot commented Mar 18, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: qingqiuhe

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hami-robot
Copy link
Copy Markdown

hami-robot bot commented Mar 18, 2026

Welcome @qingqiuhe! It looks like this is your first PR to Project-HAMi/HAMi-WebUI 🎉

@hami-robot hami-robot bot added the size/L label Mar 18, 2026
Signed-off-by: te.li <te.li@transwarp.com>
@qingqiuhe qingqiuhe force-pushed the codex/exporter-parallel-cache branch from 39d4e8c to 59cf6c0 Compare March 18, 2026 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant