[DNM] perf: use DeepCopyUpdate to eliminate Clone+DeepUpdate allocations#49762
Open
strawgate wants to merge 1 commit intoelastic:mainfrom
Open
[DNM] perf: use DeepCopyUpdate to eliminate Clone+DeepUpdate allocations#49762strawgate wants to merge 1 commit intoelastic:mainfrom
strawgate wants to merge 1 commit intoelastic:mainfrom
Conversation
botelastic
bot
added
the
needs_team
Contributor
🤖 GitHub commentsJust comment with:
|
Contributor
|
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughReplaces many ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@libbeat/processors/actions/addfields/add_fields.go`:
- Around line 140-154: The current check around af.singleKey/af.singleKeyInner
short-circuits and returns early when the top-level key exists but nested leaves
may be missing; remove or replace that shortcut so DeepCopyUpdateNoOverwrite
always runs for this case. Specifically, eliminate the early-return branch that
inspects event.Fields[af.singleKey] and returns when all immediate child keys
exist, or change it to perform a full recursive existence check against
af.singleKeyInner before returning; otherwise always call
event.Fields.DeepCopyUpdateNoOverwrite(mapstr.M{af.singleKey:
af.singleKeyInner}) so nested missing leaves (e.g., host.os.version) are merged
in.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 48d539a1-37c3-4b8e-9d6f-f57a9d3b0118
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (13)
changelog/fragments/1774757000-deepcopyupdate-allocations.yamlgo.modheartbeat/eventext/eventext.golibbeat/processors/actions/addfields/add_fields.golibbeat/processors/actions/addfields/add_fields_benchmark_test.golibbeat/processors/actions/addfields/add_fields_test.golibbeat/processors/actions/rename.golibbeat/processors/add_cloud_metadata/add_cloud_metadata.golibbeat/processors/add_cloud_metadata/add_cloud_metadata_optimize_test.golibbeat/processors/add_host_metadata/add_host_metadata.golibbeat/processors/add_host_metadata/add_host_metadata_test.golibbeat/processors/add_observer_metadata/add_observer_metadata.golibbeat/publisher/processing/default.go
strawgate
changed the title
strawgate
force-pushed
the
claude-perf-deepcopyupdate
branch
from
0072955 to
a40faea
Compare
There was a problem hiding this comment.
♻️ Duplicate comments (1)
libbeat/processors/actions/addfields/add_fields.go (1)
140-154:⚠️ Potential issue | 🔴 CriticalNested no-overwrite merge early return still incorrectly short-circuits on top-level key presence.
This check only verifies immediate child keys exist in
dstMap, butDeepCopyUpdateNoOverwritewould recursively descend into nested maps. Ifaf.singleKeyInnercontains{"os": {"version": "1.0"}}and destination has{"os": {}}, this returns early andversionis never added.,
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@libbeat/processors/actions/addfields/add_fields.go` around lines 140 - 154, The early-return incorrectly assumes presence of top-level child keys implies all nested fields exist; replace the shallow check around af.singleKey/af.singleKeyInner with a deep-existence check that recursively descends mapstr.M to verify every nested key from af.singleKeyInner exists in event.Fields[af.singleKey] (or implement a helper like deepAllKeysExist(dstMap, af.singleKeyInner) that walks nested maps and returns false if any key path is missing). Only return early when the recursive check confirms every nested path exists; otherwise call event.Fields.DeepCopyUpdateNoOverwrite(mapstr.M{af.singleKey: af.singleKeyInner}) as before.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@libbeat/processors/actions/addfields/add_fields.go`:
- Around line 140-154: The early-return incorrectly assumes presence of
top-level child keys implies all nested fields exist; replace the shallow check
around af.singleKey/af.singleKeyInner with a deep-existence check that
recursively descends mapstr.M to verify every nested key from af.singleKeyInner
exists in event.Fields[af.singleKey] (or implement a helper like
deepAllKeysExist(dstMap, af.singleKeyInner) that walks nested maps and returns
false if any key path is missing). Only return early when the recursive check
confirms every nested path exists; otherwise call
event.Fields.DeepCopyUpdateNoOverwrite(mapstr.M{af.singleKey:
af.singleKeyInner}) as before.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 64f08516-1c4b-4056-a1c1-e50c464cf010
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (13)
changelog/fragments/1774757000-deepcopyupdate-allocations.yamlgo.modheartbeat/eventext/eventext.golibbeat/processors/actions/addfields/add_fields.golibbeat/processors/actions/addfields/add_fields_benchmark_test.golibbeat/processors/actions/addfields/add_fields_test.golibbeat/processors/actions/rename.golibbeat/processors/add_cloud_metadata/add_cloud_metadata.golibbeat/processors/add_cloud_metadata/add_cloud_metadata_optimize_test.golibbeat/processors/add_host_metadata/add_host_metadata.golibbeat/processors/add_host_metadata/add_host_metadata_test.golibbeat/processors/add_observer_metadata/add_observer_metadata.golibbeat/publisher/processing/default.go
✅ Files skipped from review due to trivial changes (3)
- changelog/fragments/1774757000-deepcopyupdate-allocations.yaml
- libbeat/processors/actions/addfields/add_fields_benchmark_test.go
- libbeat/processors/actions/addfields/add_fields_test.go
🚧 Files skipped from review as they are similar to previous changes (7)
- go.mod
- libbeat/publisher/processing/default.go
- heartbeat/eventext/eventext.go
- libbeat/processors/add_cloud_metadata/add_cloud_metadata.go
- libbeat/processors/actions/rename.go
- libbeat/processors/add_host_metadata/add_host_metadata_test.go
- libbeat/processors/add_cloud_metadata/add_cloud_metadata_optimize_test.go
4 tasks
This comment has been minimized.
This comment has been minimized.
Contributor
TL;DRAll 4 failing Buildkite jobs are failing on the same Remediation
Investigation detailsRoot Cause
The failing cases are the ones that assert exact
But fixtures expect Evidence
Verification
Follow-up
Note 🔒 Integrity filtering filtered 2 itemsIntegrity filtering activated and filtered the following items during workflow execution.
What is this? | From workflow: PR Buildkite Detective Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. |
strawgate
force-pushed
the
claude-perf-deepcopyupdate
branch
from
a40faea to
296ee8c
Compare
There was a problem hiding this comment.
♻️ Duplicate comments (1)
libbeat/processors/actions/addfields/add_fields.go (1)
140-154:⚠️ Potential issue | 🟠 MajorShallow key check may skip nested merges.
The early-return at line 149 checks only immediate child key presence (
dstMap[sk]), not nested structure. Ifaf.singleKeyInnerhas{"os": {"version": "5.4"}}anddstMapalready has{"os": {"family": "linux"}}, the check seesosexists and returns — skipping the merge that would addos.version.
DeepCopyUpdateNoOverwritedescends recursively; this shortcut does not.,
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@libbeat/processors/actions/addfields/add_fields.go` around lines 140 - 154, The early-return uses a shallow presence check against dstMap (checking dstMap[sk]) which wrongly skips nested merges when af.singleKeyInner contains nested maps; update the logic around af.singleKey / af.singleKeyInner so you either remove the shallow early-return or replace it with a recursive existence check that descends maps and ensures all nested keys in af.singleKeyInner are present in the corresponding nested dstMap before returning; reference the same symbols (af.singleKey, af.singleKeyInner, dstMap, event.Fields.DeepCopyUpdateNoOverwrite) and ensure the new check mirrors DeepCopyUpdateNoOverwrite's recursion semantics so nested fields like {"os": {"version": "5.4"}} will be merged correctly instead of being skipped.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Duplicate comments:
In `@libbeat/processors/actions/addfields/add_fields.go`:
- Around line 140-154: The early-return uses a shallow presence check against
dstMap (checking dstMap[sk]) which wrongly skips nested merges when
af.singleKeyInner contains nested maps; update the logic around af.singleKey /
af.singleKeyInner so you either remove the shallow early-return or replace it
with a recursive existence check that descends maps and ensures all nested keys
in af.singleKeyInner are present in the corresponding nested dstMap before
returning; reference the same symbols (af.singleKey, af.singleKeyInner, dstMap,
event.Fields.DeepCopyUpdateNoOverwrite) and ensure the new check mirrors
DeepCopyUpdateNoOverwrite's recursion semantics so nested fields like {"os":
{"version": "5.4"}} will be merged correctly instead of being skipped.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: e91b3a1d-f5c1-4ba1-a8e7-7aa1bd280832
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (12)
changelog/fragments/1774757000-deepcopyupdate-allocations.yamlgo.modheartbeat/eventext/eventext.golibbeat/processors/actions/addfields/add_fields.golibbeat/processors/actions/addfields/add_fields_benchmark_test.golibbeat/processors/actions/addfields/add_fields_test.golibbeat/processors/add_cloud_metadata/add_cloud_metadata.golibbeat/processors/add_cloud_metadata/add_cloud_metadata_optimize_test.golibbeat/processors/add_host_metadata/add_host_metadata.golibbeat/processors/add_host_metadata/add_host_metadata_test.golibbeat/processors/add_observer_metadata/add_observer_metadata.golibbeat/publisher/processing/default.go
✅ Files skipped from review due to trivial changes (4)
- changelog/fragments/1774757000-deepcopyupdate-allocations.yaml
- libbeat/processors/add_host_metadata/add_host_metadata.go
- libbeat/processors/actions/addfields/add_fields_benchmark_test.go
- heartbeat/eventext/eventext.go
🚧 Files skipped from review as they are similar to previous changes (3)
- libbeat/processors/add_observer_metadata/add_observer_metadata.go
- libbeat/publisher/processing/default.go
- libbeat/processors/actions/addfields/add_fields_test.go
pierrehilbert
added
the
Team:Elastic-Agent-Data-Plane
Contributor
|
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
botelastic
bot
removed
the
needs_team
pierrehilbert
added
Team:obs-ds-hosted-services
Contributor
|
Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services) |
botelastic
bot
removed
the
needs_team
Replace Clone()+DeepUpdate() with single-pass DeepCopyUpdate() in add_fields, cloud/host/observer metadata, rename, processing, and heartbeat eventext. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
strawgate
force-pushed
the
claude-perf-deepcopyupdate
branch
from
4d2f151 to
8e0e2f4
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@go.mod`:
- Line 554: The go.mod currently contains a replace directive "replace
github.com/elastic/elastic-agent-libs => github.com/strawgate/elastic-agent-libs
v0.33.4-0.20260327142400-b15ccc340463"; remove that replace line from go.mod so
builds use the canonical github.com/elastic/elastic-agent-libs module, and if
you need a temporary override for local testing move the override out of the
committed go.mod (e.g., use a local go.work or documented developer step) before
merging.
In `@libbeat/processors/add_host_metadata/add_host_metadata_test.go`:
- Around line 500-556: Both tests (TestLoadDataFastPath and
TestCachedDataNotCorruptedByDownstreamMutation) must pin and restore the global
features.FQDN() flag so their behavior is deterministic: before calling
newWithHostInfoFactory set features.SetFQDN(false) (or true if you intend FQDN
on) and register a t.Cleanup to restore the previous value, then proceed to
construct the processor and run assertions; apply the same setup/cleanup pattern
in both tests so the eager cache warm-up inside newWithHostInfoFactory doesn't
depend on prior tests.
In `@libbeat/processors/add_host_metadata/add_host_metadata.go`:
- Around line 68-71: The cache TTL check uses time.Unix(0, unixNano) which
strips monotonic clock data; change the approach to use a monotonic baseline:
introduce a package-level monotonic start (e.g., startTime := time.Now()) and
store lastUpdate as elapsed monotonic nanoseconds via
time.Since(startTime).Nanoseconds() in hostMetadataCache.lastUpdate
(atomic.Int64), then compute TTL by comparing current elapsed nanoseconds
(time.Since(startTime).Nanoseconds()) against lastUpdate instead of
reconstructing a time.Time with time.Unix; update any methods that read/write
lastUpdate and the expiry check to use the new elapsed-monotonic value.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 18fbe4d6-dff0-4d2f-8dc7-3474f56531d0
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (12)
changelog/fragments/1774757000-deepcopyupdate-allocations.yamlgo.modheartbeat/eventext/eventext.golibbeat/processors/actions/addfields/add_fields.golibbeat/processors/actions/addfields/add_fields_benchmark_test.golibbeat/processors/actions/addfields/add_fields_test.golibbeat/processors/add_cloud_metadata/add_cloud_metadata.golibbeat/processors/add_cloud_metadata/add_cloud_metadata_optimize_test.golibbeat/processors/add_host_metadata/add_host_metadata.golibbeat/processors/add_host_metadata/add_host_metadata_test.golibbeat/processors/add_observer_metadata/add_observer_metadata.golibbeat/publisher/processing/default.go
✅ Files skipped from review due to trivial changes (2)
- changelog/fragments/1774757000-deepcopyupdate-allocations.yaml
- libbeat/processors/actions/addfields/add_fields.go
🚧 Files skipped from review as they are similar to previous changes (5)
- heartbeat/eventext/eventext.go
- libbeat/processors/add_observer_metadata/add_observer_metadata.go
- libbeat/publisher/processing/default.go
- libbeat/processors/actions/addfields/add_fields_benchmark_test.go
- libbeat/processors/add_cloud_metadata/add_cloud_metadata.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Replace Clone()+DeepUpdate() with single-pass DeepCopyUpdate() in add_fields, cloud/host/observer metadata, processing setup, and heartbeat eventext. DeepCopyUpdate creates fresh nested maps during merge instead of cloning the entire source first.
Per-processor benchmarks:
E2E filebeat (benchmark input → mock ES, GOMAXPROCS=2, 30s): +15.8% EPS with 6× add_fields.
Depends on elastic/elastic-agent-libs#390 (temporarily pinned to fork).
Checklist
I have made corresponding changes to the documentationI have made corresponding change to the default configuration files./changelog/fragmentsusing the changelog tool.Disruptive User Impact
None. Event output is identical to main. DeepCopyUpdate produces the same merged result as Clone()+DeepUpdate().
How to test this PR locally
Related issues