Skip to content

Add worker pool to gather pid stat#24

Merged
leborchuk merged 11 commits intomainfrom
AddWorkerPoolGatherStat
Apr 30, 2026
Merged

Add worker pool to gather pid stat#24
leborchuk merged 11 commits intomainfrom
AddWorkerPoolGatherStat

Conversation

@leborchuk
Copy link
Copy Markdown
Contributor

@leborchuk leborchuk commented Apr 28, 2026

Generate jobs for gather pid stat. Each job contain no more then JobsPerQuery pid's and belongs to a single host.

Jobs should be processed in a worker pool witch context, use for it alitto/pond - good enough to this work.

Here we add only worker pool without actual start working pool and doing some actual work.

Tests were generated. But I manually checked all of them - looks good.

Also refactor using statActivityLister - place links to it in background worker, no need to pass it as a parameter in every function.

Generate jobs for gather pid stat. Each job contain no more then JobsPerQuery pid's and belongs to a single host.

Jobs should be processed in a worker pool witch context, use for it alitto/pond - good enough to this work.

Here we add only worker pool without actual start working pool and doing some actual work.
Copilot AI review requested due to automatic review settings April 28, 2026 12:59
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds initial master-side infrastructure for collecting per-PID procfs stats by grouping session PIDs into per-host batches and executing requests via a worker pool, while also refactoring how session listing is injected into background components.

Changes:

  • Introduces GatherProcfsStat/processProcfsRequests with host grouping and pond-based worker pool submission.
  • Refactors BackgroundStorage to own a StatActivityLister dependency (and adjusts callers/tests accordingly).
  • Extracts and reuses a shared gRPC client connection cache helper in internal/master/utils.go.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 12 comments.

Show a summary per file
File Description
internal/master/utils.go Adds shared cached gRPC connection helper for master package.
internal/master/procfs.go Implements job grouping + worker-pool execution for procfs PID stat RPCs.
internal/master/procfs_test.go Adds unit tests for job grouping, request building, and pool behavior.
internal/master/deps.go Extends statActivityLister with ListAllSessions.
internal/master/background.go Moves session lister dependency into BackgroundStorage and updates refresh flow/InitBG wiring.
internal/grpc/testutils_test.go Updates test server wiring to new BackgroundStorage + master info server signatures.
internal/grpc/set_get_query_info_parallel_test.go Updates BackgroundStorage constructor usage and dialer setup.
internal/grpc/mocks_test.go Extends test mock to satisfy updated lister usage in BackgroundStorage.
internal/grpc/get_master_info.go Switches session refresh to use backgroundStorage’s lister; updates constructor signature.
internal/grpc/actions_test.go Updates BackgroundStorage constructor call site.
internal/app/app.go Wires real stat activity lister into BackgroundStorage; updates master info server construction and InitBG call.
go.mod / go.sum Adds github.com/alitto/pond dependency.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread internal/master/procfs.go Outdated
Comment thread internal/master/procfs.go Outdated
Comment thread internal/master/background.go Outdated
Comment thread internal/master/procfs.go Outdated
Comment thread internal/master/procfs.go Outdated
Comment thread internal/master/utils.go
Comment thread internal/master/procfs.go
Comment thread internal/master/procfs.go Outdated
Comment thread internal/master/procfs.go Outdated
Comment thread internal/master/procfs.go Outdated
leborchuk and others added 4 commits April 28, 2026 16:17
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI and others added 2 commits April 28, 2026 13:29
…oken loop structure in GatherProcfsStat

Agent-Logs-Url: https://github.com/open-gpdb/yagpcc/sessions/3c612e77-c1ba-4bbc-9773-cc035b3b8310

Co-authored-by: leborchuk <63977577+leborchuk@users.noreply.github.com>
Comment thread internal/grpc/deps.go Outdated
Comment thread internal/master/background.go Outdated
Comment thread internal/master/procfs.go Outdated
Comment thread internal/master/procfs.go Outdated
Comment thread internal/master/procfs.go Outdated
Comment thread internal/master/procfs.go
Comment thread internal/master/procfs.go
Comment thread internal/master/utils.go Outdated
Comment thread internal/master/background.go Outdated
Comment thread internal/master/utils.go
@leborchuk leborchuk merged commit 8b32a52 into main Apr 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants