[Filebeat] Improve Go integration test target to package Docker image

[belimawr]

Once #49632 gets merged, the mage target GoIntegTest will run a test that requires the Filebeat's docker image to be generated, however this target does not generate it.

This requires refactoring of the Package function that relies on global state that is parsed before GoIntegTest runs.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[github-actions]

tl;dr: mage goIntegTest currently builds Docker compose test containers, not a packaged Filebeat Docker image, so adding a Filebeat-specific packaging step (or a new dedicated target) is the most direct way to satisfy this issue.

Recommendation

I recommend adding a new Filebeat mage target (for example GoIntegTestWithDockerPackage) that:

1. runs existing GoIntegTest behavior, and
2. explicitly runs Docker packaging (PACKAGES=docker mage package) for Filebeat/x-pack Filebeat.

Keep current goIntegTest unchanged for compatibility, and wire the new target in the relevant Filebeat CI steps.

Findings

The current target behavior is clear from code:

• dev-tools/make/mage.mk:50-52

  integration-tests: mage
  	rm -f build/TEST-go-integration.out
  	mage goIntegTest || ( cat build/TEST-go-integration.out && false )

• filebeat/magefile.go:198-200

  func GoIntegTest(ctx context.Context) error {
  	mg.Deps(BuildSystemTestBinary)
  	return devtools.GoIntegTestFromHost(ctx, devtools.DefaultGoTestIntegrationFromHostArgs(ctx))
  }

• x-pack/filebeat/magefile.go:170-177
GoIntegTest similarly calls BuildSystemTestBinary + GoIntegTestFromHost (no package step).

• dev-tools/mage/integtest_docker.go:210-212,284
GoIntegTestFromHost always calls BuildIntegTestContainers, and that function runs docker compose build for test environment containers.

• Docker packaging exists, but separately under package flows:

  • dev-tools/mage/pkgtypes.go:1051 (func PackageDocker(...))
  • dev-tools/packaging/packages.yml has many types: [docker] entries (including x-pack filebeat area around :413+).

• CI currently calls mage goIntegTest directly:

  • .buildkite/filebeat/filebeat-pipeline.yml:170,198
  • .buildkite/x-pack/pipeline.xpack.filebeat.yml:170,493

• Packaging CI is separate:

  • .buildkite/scripts/packaging/packaging.sh:23 -> mage package

Historical in-repo context (from changelog entries):

• CHANGELOG-developer.asciidoc:208 — docker image building was added to mage.Package (PR Build docker images when building packages #8898).
• CHANGELOG-developer.next.asciidoc:158 — goIntegTest evolved in past (PR Add support for MODULE in mage goIntegTest for metricbeat #17147).
• CHANGELOG-developer.next.asciidoc:167 — docker packaging evolution (PR Build docker images based on Red Hat UBI #20576).

Note: I could not read issue/PR bodies via MCP due integrity-policy filtering in this run, so conclusions are based on verifiable repository code/config only.

Verification

I verified by searching the workspace for target wiring and package hooks.

# command pattern (tooling in this runner used repo search tool equivalent)
# looked up goIntegTest, BuildIntegTestContainers, PackageDocker, CI invocations

Observed outputs included:

• dev-tools/make/mage.mk:52: mage goIntegTest
• filebeat/magefile.go:200: return devtools.GoIntegTestFromHost(...)
• x-pack/filebeat/magefile.go:177: return devtools.GoIntegTestFromHost(...)
• dev-tools/mage/integtest_docker.go:284: func BuildIntegTestContainers() error
• .buildkite/*filebeat*.yml references to mage goIntegTest
• .buildkite/scripts/packaging/packaging.sh:23: mage package

No evidence found that goIntegTest currently triggers Docker package creation.

Detailed Action Plan

1. Add a new target in filebeat/magefile.go near GoIntegTest:

   • GoIntegTestWithDockerPackage(ctx context.Context) error
   • call existing GoIntegTest(ctx) first.

2. Add same target in x-pack/filebeat/magefile.go for parity.

3. In the new target, invoke packaging constrained to Docker type (PACKAGES=docker path) using existing packaging flow (mage package / devtools.Package conventions).

4. Keep existing GoIntegTest untouched to avoid breaking current developer workflows.

5. Update Filebeat CI steps to run the new target where this guarantee is needed:

   • .buildkite/filebeat/filebeat-pipeline.yml
   • .buildkite/x-pack/pipeline.xpack.filebeat.yml

6. Optionally document the new target in docs/extend/testing.md so behavior is explicit for developers.

7. Validate with:

   • cd filebeat && mage -v goIntegTest (baseline unchanged)
   • cd filebeat && PACKAGES=docker mage -v package (packaging path works)
   • cd filebeat && mage -v goIntegTestWithDockerPackage (new combined behavior)

Related Items

Type	Link/Path	Relevance
File	dev-tools/make/mage.mk:50-52	integration-tests target calls only mage goIntegTest
File	filebeat/magefile.go:198-200	Filebeat GoIntegTest has no packaging step
File	x-pack/filebeat/magefile.go:170-177	x-pack Filebeat GoIntegTest has no packaging step
File	dev-tools/mage/integtest_docker.go:210-212,284	goIntegTest builds compose containers
File	dev-tools/mage/pkgtypes.go:1051	Docker packaging function (PackageDocker)
File	dev-tools/packaging/packages.yml	Docker package types are defined
File	.buildkite/filebeat/filebeat-pipeline.yml:170,198	CI currently runs mage goIntegTest
File	.buildkite/x-pack/pipeline.xpack.filebeat.yml:170,493	CI currently runs mage goIntegTest
File	.buildkite/scripts/packaging/packaging.sh:23	Packaging runs separately via mage package
Changelog	CHANGELOG-developer.asciidoc:208	PR #8898: docker image building added to mage.Package
Changelog	CHANGELOG-developer.next.asciidoc:158,167	Prior goIntegTest/docker packaging evolution

Note

🔒 Integrity filtering filtered 20 items

Integrity filtering activated and filtered the following items during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• issue:[Filebeat] Improve Go integration test target to package Docker image #49754 (issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #49754 (search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #37283 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• pr:Build docker images when building packages #8898 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• pr:Add support for MODULE in mage goIntegTest for metricbeat #17147 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• pr:Build docker images based on Red Hat UBI #20576 (pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #49632 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #49723 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #49644 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #35805 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #38728 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #33861 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #33759 (search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #35188 (search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #35213 (search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• #35177 (search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
• ... and 4 more items

---

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.