sei: Update packages to format_version 3.0.0

[andrewkroh]

This meta issue tracks the work related to updating packages to format_version: 3.0.0.

• @kgeller Fix Invalid ECS field usages at root-level #7808
• Remove version from ingest pipelines - [bitdefender,forcepoint_web,jumpcloud,tines] Remove version from ingest pipeline definitions #7807
• hashicorp_vault.metrics - Resolve errors related to the dynamic mapping fields - [hashicorp_vault] Convert dynamic mappings to fields.yml #7811
• Add DLM to packages that have ILM
  • ti_maltiverse [ti_maltiverse] Add DLM policy and Update format_version to 3.0.0 #7851
  • ti_anomali [ti_anomali] Add DLM policy and Update format_version to 3.0.0 #7849
  • ti_recorded_future [ti_recordedfuture] Add DLM policy and update format_version to 3.0.0 #7848
  • ti_rapid7 [ti_rapid7] Update format_version to 3.0.0 #7910
• Fix dotted YAML keys
  • [auditd] Remove elasticsearch.dynamic_{dataset,namespace} #7800
  • [ti_maltiverse] Remove dotted yaml keys #7804
  • [ti_anomali] Remove dotted-yaml keys #7803
  • [ti_recordedfuture] Remove dotted-yaml keys #7801
  • The remaining keys (all under constraints) will be handled in bulk via ecs-update.
• @marc-gr Change format_version 3.0.0 Update package spec to v3.0 for SEI owned integrations #7883
  • This will be handled in bulk via ecs-update.
  • We also need to include owner.type: elastic as part of these changes (see: [Epic] Tagging Community/Partner Integrations #6569)
• Integrations requiring manual tests: [SEI] Integrations requiring manual testing for spec 3.0 #7814
• No elastic.capabilities constraints should be added. This will ensure our packages are available across observability and security projects.

The general command to update packages in bulk is:

go run github.com/andrewkroh/go-examples/ecs-update@main -format-version=3.0.0 -fix-dotted-yaml-keys -owner elastic/security-external-integrations packages/*

but we want to exclude non-deprecated rsa2elk packages from using format_version 3.0.0 so use this command and glob:

zsh
setopt extendedglob
go run github.com/andrewkroh/go-examples/ecs-update@main  \
  -format-version=3.0.0 \
  -fix-dotted-yaml-keys \
  -add-owner-type \
  -owner elastic/security-external-integrations \
  packages/*~packages/cylance~packages/fortinet_forticlient~packages/imperva~packages/netscout~packages/radware~packages/squid

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

116 of the SEI owned packages can be changed to 3.0.0 without issue. The 24 below have some kind of issue that needs resolved. This does account for the DLM changes.

• 1password
• azure_frontdoor
• carbonblack_edr
• cisco_aironet
• cisco_meraki
• cloudflare_logpush
• crowdstrike
• cylance
• fireeye
• fortinet_forticlient
• hashicorp_vault
• imperva
• infoblox_nios
• juniper_srx
• lyve_cloud
• netflow
• netscout
• panw
• radware
• sentinel_one
• squid
• ti_anomali
• ti_maltiverse
• ti_rapid7
• ti_recorded_future
• trend_micro_vision_one

[P1llus]

Added #7814 to the description as well.

[ebeahan]

No elastic.capabilities constraints should be added. This will ensure our packages are available across observability and security projects.

As we're testing for this update and looking beyond, do we consider having testing in place for both Observability and Security project types?

such as:

# provision security type
elastic-package stack up -v --version 8.10.1 --provider serverless -U stack.serverless.type=security
# test
elastic-package test -v
# spin it down
elastic-package stack down -v
# repeat with observability project
elastic-package stack up -v --version 8.10.1 --provider serverless -U stack.serverless.type=observability
elastic-package test -v
elastic-package stack down -v

[andrewkroh]

After #8025 merges all of the packages owned by SEI (minus deprecated and rsa2elk) will be updated to use format_version 3.0.0.