Skip to content

Add Elastic Agent metadata to the org-data command#347

Open
opauloh wants to merge 5 commits intomainfrom
gen-org-elastic-agent
Open

Add Elastic Agent metadata to the org-data command#347
opauloh wants to merge 5 commits intomainfrom
gen-org-elastic-agent

Conversation

@opauloh
Copy link
Copy Markdown
Collaborator

@opauloh opauloh commented Apr 2, 2026

Updates the organization data model and integration logic to include stable Elastic Agent identification in all generated documents. This ensures that synthetic data correctly mimics real-world ECS-compliant logs and supports security features that rely on agent-based correlation.

It generates per agents per workstation for Endpoint integration and a central agent for SaaS integrations, simulating real world environment.

Key changes:

  • Adds elasticAgentId to Device and Host models, and centralAgent to the Organization model.
  • Implements buildCentralAgent, buildLocalAgent, and buildServerAgent helpers in BaseIntegration.
  • Updates all existing integrations to include the agent field in their documents.

Additional changes

  • Expands the Okta system integration with anomalous event generation to support testing for Post-Authentication Detection (PAD).

opauloh added 2 commits April 1, 2026 14:43
Updates the organization data model and integration logic to include stable Elastic Agent identification in all generated documents. This ensures that synthetic data correctly mimics real-world ECS-compliant logs and supports security features that rely on agent-based correlation.

Key changes:
- Adds `elasticAgentId` to Device and Host models, and `centralAgent` to the Organization model.
- Implements `buildCentralAgent`, `buildLocalAgent`, and `buildServerAgent` helpers in `BaseIntegration`.
- Updates all existing integrations to include the `agent` field in their documents.
- Expands the Okta system integration with anomalous event generation to support testing for Post-Authentication Detection (PAD).
@opauloh opauloh requested a review from a team as a code owner April 2, 2026 21:47
@opauloh opauloh requested review from Copilot and tiansivive April 2, 2026 21:47
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the org-data synthetic data generator to attach ECS agent.* metadata to all generated documents, using stable identifiers for local workstation agents, server agents, and a single central collector agent for SaaS/cloud integrations. It also expands the Okta System Logs integration with additional org-level and anomalous event generation to better exercise Post-Authentication Detection (PAD) scenarios.

Changes:

  • Adds elasticAgentId to Device and Host, plus centralAgent to Organization, and generates these values in the org data generator.
  • Introduces buildCentralAgent, buildLocalAgent, and buildServerAgent helpers (and AgentData) in BaseIntegration, then wires agent into integration documents across many integrations.
  • Enhances okta_system event generation with org-level lifecycle/privilege/app assignment events and anomalous PAD-oriented patterns.

Reviewed changes

Copilot reviewed 54 out of 54 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
src/commands/org_data/types.ts Adds agent-related fields/types (elasticAgentId, CentralAgent, Organization.centralAgent).
src/commands/org_data/org_data_generator.ts Generates centralAgent, plus elasticAgentId for devices/hosts.
src/commands/org_data/integrations/base_integration.ts Adds AgentData, ELASTIC_AGENT_VERSION, and agent builder helpers for central/local/server agents.
src/commands/org_data/integrations/endpoint_integration.ts Switches to device.elasticAgentId, standardizes agent version, and adds hostname to agent object.
src/commands/org_data/integrations/system_integration.ts Adds server agent metadata via buildServerAgent(host) in emitted docs.
src/commands/org_data/integrations/zscaler_zia_integration.ts Adds local agent metadata to web/firewall docs (currently conditional).
src/commands/org_data/integrations/jamf_pro_integration.ts Adds local agent metadata tied to workstation hostname.
src/commands/org_data/integrations/island_browser_integration.ts Adds central agent to SaaS docs and local agent to device/audit docs.
src/commands/org_data/integrations/crowdstrike_integration.ts Adds local/central agent metadata depending on Falcon event type and document.
src/commands/org_data/integrations/okta_integration.ts Adds central agent to sync marker and entity docs.
src/commands/org_data/integrations/okta_system_integration.ts Adds central agent to emitted documents and expands PAD-oriented event generation.
src/commands/org_data/integrations/entra_id_integration.ts Adds central agent to sync marker and entity docs.
src/commands/org_data/integrations/active_directory_integration.ts Adds central agent to AD user/computer docs.
src/commands/org_data/integrations/cloudtrail_integration.ts Threads org into event builders and adds central agent to CloudTrail docs.
src/commands/org_data/integrations/azure_integration.ts Threads a shared central agent through Azure log builders and adds agent to emitted docs.
src/commands/org_data/integrations/gcp_integration.ts Adds central agent to GCP audit/firewall docs.
src/commands/org_data/integrations/cloud_asset_integration.ts Adds central agent to asset documents.
src/commands/org_data/integrations/cloudflare_logpush_integration.ts Adds central agent to Cloudflare Logpush http/firewall docs.
src/commands/org_data/integrations/google_workspace_integration.ts Adds central agent to Google Workspace docs.
src/commands/org_data/integrations/github_integration.ts Adds central agent to GitHub audit docs.
src/commands/org_data/integrations/gitlab_integration.ts Adds central agent to GitLab audit/api/auth docs.
src/commands/org_data/integrations/slack_integration.ts Adds central agent to Slack audit docs and threads through helpers.
src/commands/org_data/integrations/zoom_integration.ts Adds central agent to Zoom webhook docs and threads through helper signature.
src/commands/org_data/integrations/workday_integration.ts Adds central agent to Workday docs and threads through helper signature.
src/commands/org_data/integrations/servicenow_integration.ts Adds central agent to incident/change docs and threads through helper signatures.
src/commands/org_data/integrations/ti_abusech_integration.ts Adds central agent to TI docs and threads through helper signatures.
src/commands/org_data/integrations/thycotic_ss_integration.ts Adds central agent to Thycotic Secret Server docs and threads through helper signature.
src/commands/org_data/integrations/teleport_integration.ts Adds central agent to Teleport audit docs and threads through helper signature.
src/commands/org_data/integrations/onepassword_integration.ts Adds central agent to 1Password docs.
src/commands/org_data/integrations/o365_integration.ts Adds central agent to O365 audit docs.
src/commands/org_data/integrations/mongodb_atlas_integration.ts Adds central agent to MongoDB Atlas docs and threads through helper signatures.
src/commands/org_data/integrations/mattermost_integration.ts Adds central agent to Mattermost audit docs and threads through helper signature.
src/commands/org_data/integrations/lyve_cloud_integration.ts Adds central agent to Lyve Cloud audit docs and threads through helper signature.
src/commands/org_data/integrations/lastpass_integration.ts Adds central agent to user/event/shared-folder docs and threads through helper signatures.
src/commands/org_data/integrations/keycloak_integration.ts Adds central agent to Keycloak log docs and threads through helper signature.
src/commands/org_data/integrations/keeper_integration.ts Adds central agent to Keeper audit docs and threads through helper signature.
src/commands/org_data/integrations/jumpcloud_integration.ts Adds central agent to JumpCloud events and threads through helper signature.
src/commands/org_data/integrations/hashicorp_vault_integration.ts Adds central agent to Vault audit/log docs and threads through helper signatures.
src/commands/org_data/integrations/forgerock_integration.ts Adds central agent to ForgeRock docs and threads through helper signatures.
src/commands/org_data/integrations/cyberark_pas_integration.ts Adds central agent to CyberArk PAS audit docs and threads through helper signature.
src/commands/org_data/integrations/cisco_duo_integration.ts Adds central agent to Duo auth docs.
src/commands/org_data/integrations/canva_integration.ts Adds central agent to Canva audit docs and threads through helper signature.
src/commands/org_data/integrations/box_integration.ts Adds central agent to Box event docs and threads through helper signature.
src/commands/org_data/integrations/bitwarden_integration.ts Adds central agent to Bitwarden docs and threads through helper signatures.
src/commands/org_data/integrations/beyondinsight_integration.ts Adds central agent to BeyondInsight docs and threads through helper signatures.
src/commands/org_data/integrations/auth0_integration.ts Adds central agent to Auth0 logs and threads through helper signature.
src/commands/org_data/integrations/authentik_integration.ts Adds central agent to Authentik user/group/event docs and threads through helper signatures.
src/commands/org_data/integrations/atlassian_jira_integration.ts Adds central agent to Jira audit docs and threads through helper signature.
src/commands/org_data/integrations/atlassian_confluence_integration.ts Adds central agent to Confluence audit docs and threads through helper signature.
src/commands/org_data/integrations/atlassian_bitbucket_integration.ts Adds central agent to Bitbucket audit docs and threads through helper signature.
src/commands/org_data/integrations/sailpoint_integration.ts Adds central agent to SailPoint event docs and threads through helper signature.
src/commands/org_data/integrations/ping_one_integration.ts Adds central agent to PingOne audit docs and threads through helper signature.
src/commands/org_data/integrations/ping_directory_integration.ts Adds central agent to PingDirectory SCIM docs and threads through helper signature.
.agents/skills/update-org-data-integrations/SKILL.md Documents the new stable agent fields and correlation rules.

Comment on lines +115 to 119
const hostname = laptop ? `${employee.userName}-${laptop.platform}` : 'unknown';
return {
'@timestamp': timestamp,
agent: laptop ? this.buildLocalAgent(laptop, hostname) : undefined,
message: JSON.stringify({ event: rawEvent }),
Copy link

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agent is conditionally set to undefined when no laptop is found, but org generation always creates at least one laptop per employee. This conditional adds dead code and can silently omit agent (since undefined fields drop during JSON serialization). Consider removing the fallback and always setting agent via buildLocalAgent(...) to keep documents ECS-consistent.

Copilot uses AI. Check for mistakes.
Comment on lines +1231 to +1233
console.log(
` Generating PAD anomalous Okta patterns for ${rogueEmployees.length} rogue actor(s)...`,
);
Copy link

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoid using console.log here; it will spam stdout during generation and bypasses the repo’s structured logger. Use log.info (or a debug-level log) from src/utils/logger.ts, or remove these statements entirely.

Copilot uses AI. Check for mistakes.
Comment on lines +1294 to +1296
// Replace the membership events with properly attributed ones
events.length = events.length - membershipCount;
for (let i = 0; i < membershipCount; i++) {
Copy link

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code first pushes membershipCount group membership events and then immediately removes them by truncating the array, only to regenerate the same count with createAdminActionEvent. This wastes work and advances Faker’s RNG unnecessarily. Consider generating only the attributed events (second loop) and deleting the first loop + manual truncation.

Copilot uses AI. Check for mistakes.
Comment on lines 138 to +142
private createWebhookDocument(
employee: Employee,
org: Organization,
masterAccountId: string,
centralAgent: { id: string; name: string; type: string; version: string },
Copy link

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The centralAgent parameter type is duplicated inline here. To avoid drift and keep signatures consistent across integrations, consider importing and using the shared AgentData type from base_integration.ts instead of redefining { id; name; type; version }.

Copilot uses AI. Check for mistakes.
opauloh added 3 commits April 10, 2026 15:56
Resolve conflicts in azure_integration.ts by combining main's improved
audit-log logic (distinct target employees, user-user relationship
enrichment, always user-initiated) with the branch's centralAgent
metadata propagation. Fix duplicate hostname declaration in
jamf_pro_integration.ts introduced by the auto-merge.

Made-with: Cursor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants