Add helper methods to support kafka output in beatreceivers

[khushijain21]

Proposed commit message

This PR adds helper methods on fmtstr package to help parse dynamic fields. This is required to support kafka output on beatreceivers

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None

Related issues

• Relates https://github.com/elastic/ingest-dev/issues/7162

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @khushijain21? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The PR exports previously internal symbols and refactors token parsing. parseEventPath was renamed to ParseEventPath. makeLexer was renamed to MakeLexer and gained a bounds check and corrected two-byte operator slicing. Parsing was refactored into a generic parseFormatTokens loop with a new parseVariableToken helper. A new exported type VariableToken and function ParseRawTokens were added. Tests and a fuzz test for ParseRawTokens were included.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

• 🛠️ Update Documentation: Commit on current branch
• 🛠️ Update Documentation: Create PR

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@libbeat/common/fmtstr/formatstring.go`:
- Around line 340-365: ParseRawTokens can return early and leave the lexer
producer goroutine blocked; ensure we always drain lex.Tokens() on any early
exit by adding a deferred drain. At the top of ParseRawTokens, add a defer that
iterates over lex.Tokens() and discards remaining tokens (for range lex.Tokens()
{ }) unless the function completes normally; mark normal completion (e.g., a
local done flag set to true just before the final return) so the deferred drain
only runs on error/early returns. This ensures any early returns from cases like
tokErr or parseVariableToken failures do not leak the goroutine started by
MakeLexer while keeping normal behavior for the existing token handling
(tokString, tokOpen, tokClose, tokOperator) and existing use of
parseVariableToken and VariableToken.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6a04693a-87bf-405a-8074-0a210fd48f5c

📥 Commits

Reviewing files that changed from the base of the PR and between 2d10f57 and f53f390.

📒 Files selected for processing (3)

• libbeat/common/fmtstr/formatevents.go
• libbeat/common/fmtstr/formatstring.go
• libbeat/common/fmtstr/formatstring_test.go

[belimawr]

Overall LGMT, but I agree with Mauri, a more exhaustive test for ParseRawTokens would be great.

[coderabbitai]

♻️ Duplicate comments (1)

libbeat/common/fmtstr/formatstring.go (1)

351-361: ⚠️ Potential issue | 🟠 Major

Goroutine leak: ParseRawTokens must drain lexer on early exit.

MakeLexer spawns a goroutine. If parsing fails mid-stream (e.g., errNestedVar at line 378), this function returns without draining lex.Tokens(), blocking the producer indefinitely.

Proposed fix

 func ParseRawTokens(lex lexer) ([]any, error) {
+	defer lex.Finish()
 	return parseFormatTokens(lex, func(elems *[]any, s string) {

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@libbeat/common/fmtstr/formatstring.go` around lines 351 - 361, ParseRawTokens
can return early while MakeLexer’s producer goroutine is still sending tokens,
causing a goroutine leak; ensure the lexer channel is drained on any early
return. Change ParseRawTokens so after calling parseFormatTokens it checks for a
non-nil error and if present iterates over lex.Tokens() to exhaust the channel
(e.g., for range lex.Tokens() {} ) before returning the error; reference
functions: ParseRawTokens, parseFormatTokens and the lex.Tokens() method, and
preserve existing return values (including wrapping/returning the original
error).

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@libbeat/common/fmtstr/formatstring.go`:
- Around line 351-361: ParseRawTokens can return early while MakeLexer’s
producer goroutine is still sending tokens, causing a goroutine leak; ensure the
lexer channel is drained on any early return. Change ParseRawTokens so after
calling parseFormatTokens it checks for a non-nil error and if present iterates
over lex.Tokens() to exhaust the channel (e.g., for range lex.Tokens() {} )
before returning the error; reference functions: ParseRawTokens,
parseFormatTokens and the lex.Tokens() method, and preserve existing return
values (including wrapping/returning the original error).

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 23c8ab50-cf3a-4536-b190-bc8c098334cd

📥 Commits

Reviewing files that changed from the base of the PR and between f53f390 and b9bbdce.

📒 Files selected for processing (2)

• libbeat/common/fmtstr/formatstring.go
• libbeat/common/fmtstr/formatstring_test.go

✅ Files skipped from review due to trivial changes (1)

• libbeat/common/fmtstr/formatstring_test.go

[coderabbitai]

♻️ Duplicate comments (1)

libbeat/common/fmtstr/formatstring.go (1)

351-361: ⚠️ Potential issue | 🔴 Critical

Drain lexer on ParseRawTokens error paths to prevent goroutine leaks.

At Line 351, ParseRawTokens can return early (malformed input) without draining lex.Tokens(). Because MakeLexer uses a producer goroutine, this can leak blocked goroutines.

Proposed fix

func ParseRawTokens(lex lexer) ([]any, error) {
+	defer lex.Finish()
	return parseFormatTokens(lex, func(elems *[]any, s string) {
		*elems = append(*elems, s)
	}, func(lex lexer) (any, error) {

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@libbeat/common/fmtstr/formatstring.go` around lines 351 - 361, ParseRawTokens
can return early leaving the lexer producer goroutine blocked; when
parseFormatTokens returns an error you must start a background goroutine to
fully drain lex.Tokens() before returning to avoid leaking the producer. Modify
ParseRawTokens so after calling parseFormatTokens you check the returned error
and if non-nil spawn a goroutine that ranges over lex.Tokens() (for range
lex.Tokens() { }) to consume remaining tokens, then return the error; reference
the ParseRawTokens function, the lex parameter (type lexer) and lex.Tokens()
channel and ensure this drain happens on all error return paths from
ParseRawTokens.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@libbeat/common/fmtstr/formatstring.go`:
- Around line 351-361: ParseRawTokens can return early leaving the lexer
producer goroutine blocked; when parseFormatTokens returns an error you must
start a background goroutine to fully drain lex.Tokens() before returning to
avoid leaking the producer. Modify ParseRawTokens so after calling
parseFormatTokens you check the returned error and if non-nil spawn a goroutine
that ranges over lex.Tokens() (for range lex.Tokens() { }) to consume remaining
tokens, then return the error; reference the ParseRawTokens function, the lex
parameter (type lexer) and lex.Tokens() channel and ensure this drain happens on
all error return paths from ParseRawTokens.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2d66f647-ac32-4cd9-bce7-d96960240585

📥 Commits

Reviewing files that changed from the base of the PR and between b9bbdce and 39b86ef.

📒 Files selected for processing (2)

• libbeat/common/fmtstr/formatstring.go
• libbeat/common/fmtstr/formatstring_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• libbeat/common/fmtstr/formatstring_test.go

[github-actions]

TL;DR

Buildkite failed because check-no-changes detected an uncommitted update to libbeat/common/fmtstr/formatstring_test.go. Regenerate/update and commit that file (likely formatting drift) before re-running CI.

Remediation

• On branch khushijain21:dynamic-topic-kafka, run:
  • make -C libbeat check update
  • gofmt -w libbeat/common/fmtstr/formatstring_test.go (if still dirty)
  • make check-no-changes
• Commit the resulting file changes, then re-run the Buildkite pipeline.

Investigation details

Root Cause

libbeat check/update stage exits because the repo is left dirty after update/check. The failing step reports one modified file:

• libbeat/common/fmtstr/formatstring_test.go

The failure is enforced by:

• dev-tools/mage/check.go:66-68 (some files are not up-to-date. Run 'make update'...)
• Makefile:111-116 (check-no-changes uses git diff-index --exit-code HEAD --)

Evidence

• Build: https://buildkite.com/elastic/beats/builds/43355
• Job/step: Libbeat: Run check/update (beats-libbeat build 27676)
• Key log excerpt (/tmp/gh-aw/buildkite-logs/beats-libbeat-libbeat-run-checkupdate.txt):

Error: some files are not up-to-date. Run 'make update' then review and commit the changes. Modified: [libbeat/common/fmtstr/formatstring_test.go]
make: *** [scripts/Makefile:141: check] Error 1

Verification

• I reviewed the failing Buildkite log and repository check enforcement code paths.
• Full local make -C libbeat check update was not runnable in this environment because mage is unavailable in PATH here, so CI log evidence was used as source of truth.

Follow-up

If the file still changes after gofmt + make update, confirm the Go toolchain version used locally matches CI to avoid formatter/version drift.

Note

🔒 Integrity filtering filtered 1 item

Integrity filtering activated and filtered the following item during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• pr:Add helper methods to support kafka output in beatreceivers #49768 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.