From 43867469e972c2c350f7d41bbd19b11226c27efc Mon Sep 17 00:00:00 2001 From: May Lee Date: Thu, 30 Oct 2025 17:05:22 -0400 Subject: [PATCH 1/3] add packs --- .../observability_pipelines/packs/_index.md | 70 +++++++++++++++++++ .../packs/akamai_cdn.md | 18 +++++ .../packs/amazon_cloudfront.md | 19 +++++ .../packs/amazon_vpc_flow_logs.md | 19 +++++ .../packs/aws_cloudtrail.md | 19 +++++ .../packs/cisco_asa.md | 19 +++++ .../packs/cloudflare.md | 19 +++++ .../en/observability_pipelines/packs/f5.md | 19 +++++ .../observability_pipelines/packs/fastly.md | 19 +++++ .../packs/fortinet_firewall.md | 19 +++++ .../packs/haproxy_ingress.md | 19 +++++ .../packs/istio_proxy.md | 19 +++++ .../observability_pipelines/packs/netskope.md | 19 +++++ .../en/observability_pipelines/packs/nginx.md | 19 +++++ .../en/observability_pipelines/packs/okta.md | 19 +++++ .../packs/palo_alto_firewall.md | 19 +++++ .../packs/windows_xml.md | 19 +++++ .../packs/zscaler_zia_dns.md | 19 +++++ .../packs/zscaler_zia_firewall.md | 19 +++++ .../packs/zscaler_zia_tunnel.md | 19 +++++ .../packs/zscaler_zia_web_logs.md | 19 +++++ 21 files changed, 449 insertions(+) create mode 100644 content/en/observability_pipelines/packs/_index.md create mode 100644 content/en/observability_pipelines/packs/akamai_cdn.md create mode 100644 content/en/observability_pipelines/packs/amazon_cloudfront.md create mode 100644 content/en/observability_pipelines/packs/amazon_vpc_flow_logs.md create mode 100644 content/en/observability_pipelines/packs/aws_cloudtrail.md create mode 100644 content/en/observability_pipelines/packs/cisco_asa.md create mode 100644 content/en/observability_pipelines/packs/cloudflare.md create mode 100644 content/en/observability_pipelines/packs/f5.md create mode 100644 content/en/observability_pipelines/packs/fastly.md create mode 100644 content/en/observability_pipelines/packs/fortinet_firewall.md create mode 100644 content/en/observability_pipelines/packs/haproxy_ingress.md create mode 100644 content/en/observability_pipelines/packs/istio_proxy.md create mode 100644 content/en/observability_pipelines/packs/netskope.md create mode 100644 content/en/observability_pipelines/packs/nginx.md create mode 100644 content/en/observability_pipelines/packs/okta.md create mode 100644 content/en/observability_pipelines/packs/palo_alto_firewall.md create mode 100644 content/en/observability_pipelines/packs/windows_xml.md create mode 100644 content/en/observability_pipelines/packs/zscaler_zia_dns.md create mode 100644 content/en/observability_pipelines/packs/zscaler_zia_firewall.md create mode 100644 content/en/observability_pipelines/packs/zscaler_zia_tunnel.md create mode 100644 content/en/observability_pipelines/packs/zscaler_zia_web_logs.md diff --git a/content/en/observability_pipelines/packs/_index.md b/content/en/observability_pipelines/packs/_index.md new file mode 100644 index 0000000000000..e2a2b32be978a --- /dev/null +++ b/content/en/observability_pipelines/packs/_index.md @@ -0,0 +1,70 @@ +--- +title: Packs +description: Learn more about Observability Pipelines Packs +disable_toc: false +--- + +## Overview + +When you setup a pipeline to send a logs from a specific source to Observability Pipelines, you likely have to think about questions such as: + +- Which logs from this source are important? +- Which logs from this source should I drop? +- Which logs should I retain? +- Should I sample logs? +- Should I add quotas? + +Oftentimes you also have to talk to different teams to find answers to these questions. + +Observability Pipelines Packs are predefined configurations that help you set up and optimize Observability Pipelines without extensive manual configuration. Each pack is specific to a source and identifies: + +- Log fields that can safely be removed from the logs. +- Logs that can be dropped, such as duplicated logs. +- Logs that need to be parsed. +- Logs that need to be formatted for the destination. + +## Setup + +To set up packs: + +1. Navigate to the [Pipelines][1] page. +1. Click **Packs**. +1. Click the pack you want to set up. +1. You can either create a new pipeline from the pack or add the pack to an existing pipelines. +- If you clicked **Add to New Pipeline**, the pack is added to a new pipeline. + - Click the processor group that was added to see the individual processors the pack added and edit them as needed. See [Processors][2] for more information. + - See [Set Up Pipelines][3] for information on setting up the pipeline. +- If you clicked **Add to Existing Pipeline**: + 1. Select the pipeline you want to add the pack to. + 1. Click **Add to Existing Pipeline**. + - The pack is added to the last processor group in your pipeline. + - Click on the group to review the individual processors and edit them as needed. See [Processors][2] for more information. + +## Packs + +These are the available packs: + +- Akamai CDN +- AWS CloudFront +- AWS CloudTrail +- Amazon VPC Flow Logs +- Cisco ASA +- Cloudflare +- F5 +- Fastly +- Fortinet Firewall +- HAProxy Ingress +- Istio Proxy +- Netskope +- NGINX +- Okta +- Palo Alto Firewall +- Windows XML +- ZScaler ZIA DNS +- Zscaler ZIA Firewall +- Zscaler ZIA Tunnel +- Zscaler ZIA Web Logs + +[1]: https://app.datadoghq.com/observability-pipelines +[2]: /observability_pipelines/processors/ +[3]: /observability_pipelines/set_up_pipelines/ \ No newline at end of file diff --git a/content/en/observability_pipelines/packs/akamai_cdn.md b/content/en/observability_pipelines/packs/akamai_cdn.md new file mode 100644 index 0000000000000..c85fe9b11effe --- /dev/null +++ b/content/en/observability_pipelines/packs/akamai_cdn.md @@ -0,0 +1,18 @@ +--- +title: Akamai CDN +description: Learn more about the Akamai CDN pack. +--- + +## Overview + +Akamai logs show client requests and responses at the edge. + +What this pack does: + +- Drops static files, health checks, and bot noise +- Keeps failed, risky, and unusual requests +- Normalizes key values for analysis + +## Example logs + +TKTK \ No newline at end of file diff --git a/content/en/observability_pipelines/packs/amazon_cloudfront.md b/content/en/observability_pipelines/packs/amazon_cloudfront.md new file mode 100644 index 0000000000000..685b6282e0696 --- /dev/null +++ b/content/en/observability_pipelines/packs/amazon_cloudfront.md @@ -0,0 +1,19 @@ +--- +title: AWS CloudFront +description: Learn more about the AWS CloudFront pack. +--- + +## Overview + +AWS CloudFront logs show requests, cache use, and edge activity + +What this pack does: + +- Deduplicates repeated request logs +- Samples high-volume cache hits +- Drops health checks, static, and bot logs + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/amazon_vpc_flow_logs.md b/content/en/observability_pipelines/packs/amazon_vpc_flow_logs.md new file mode 100644 index 0000000000000..b6975b5411d33 --- /dev/null +++ b/content/en/observability_pipelines/packs/amazon_vpc_flow_logs.md @@ -0,0 +1,19 @@ +--- +title: Amazon VPC Flow Logs +description: Learn more about the Amazon VPC Flow Logs pack. +--- + +## Overview + +Amazon VPC Flow Logs capture network traffic between VPC resources. + +What this pack does: + +- Removes unused log metadata +- Drops idle and internal `ACCEPT OK` flows +- Keeps denied and rejected connections + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/aws_cloudtrail.md b/content/en/observability_pipelines/packs/aws_cloudtrail.md new file mode 100644 index 0000000000000..76d67d7bb998c --- /dev/null +++ b/content/en/observability_pipelines/packs/aws_cloudtrail.md @@ -0,0 +1,19 @@ +--- +title: AWS CloudTrail +description: Learn more about the AWS CloudTrail pack. +--- + +## Overview + +AWS CloudTrail records API calls and account activity across AWS services. + +What this pack does: + +- Samples high-frequency, read-only API calls +- Splits multi-record events into clean entries +- Scans for keys, credentials, and role IDs + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/cisco_asa.md b/content/en/observability_pipelines/packs/cisco_asa.md new file mode 100644 index 0000000000000..63f58c44b0e56 --- /dev/null +++ b/content/en/observability_pipelines/packs/cisco_asa.md @@ -0,0 +1,19 @@ +--- +title: Cisco ASA +description: Learn more about the Cisco ASA pack. +--- + +## Overview + +Cisco ASA firewall logs capture syslog events for traffic, VPNs, and security alerts. + +What this pack does: + +- Drops redundant and low-value logs +- Normalizes ASA codes +- Keeps denied, VPN, and high alerts + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/cloudflare.md b/content/en/observability_pipelines/packs/cloudflare.md new file mode 100644 index 0000000000000..34855ac16e783 --- /dev/null +++ b/content/en/observability_pipelines/packs/cloudflare.md @@ -0,0 +1,19 @@ +--- +title: Cloudflare +description: Learn more about the Cloudflare pack. +--- + +## Overview + +Cloudflare logs show edge traffic, performance, and security. + +What this pack does: + +- Deduplicates traffic, samples low-risk logs +- Keeps key requests and security events +- Filters secrets and removes sensitive fields + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/f5.md b/content/en/observability_pipelines/packs/f5.md new file mode 100644 index 0000000000000..7ebd1040a8d15 --- /dev/null +++ b/content/en/observability_pipelines/packs/f5.md @@ -0,0 +1,19 @@ +--- +title: F5 +description: Learn more about the F5 pack. +--- + +## Overview + +F5 logs capture traffic, security policy, and intrusion events. + +What this pack does: + +- Keeps blocked, denied, and intrusion alerts +- Samples high-volume traffic categories +- Drops routine allow events and health checks + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/fastly.md b/content/en/observability_pipelines/packs/fastly.md new file mode 100644 index 0000000000000..4121d14ce8491 --- /dev/null +++ b/content/en/observability_pipelines/packs/fastly.md @@ -0,0 +1,19 @@ +--- +title: Fastly +description: Learn more about the Fastly pack. +--- + +## Overview + +Fastly CDN logs record client requests, cache states, and delivery performance. + +What this pack does: + +- Samples static assets and `200 OK`s +- Keeps cache errors and slow outliers +- Normalizes fields, drops bulky headers + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/fortinet_firewall.md b/content/en/observability_pipelines/packs/fortinet_firewall.md new file mode 100644 index 0000000000000..45ebd457af004 --- /dev/null +++ b/content/en/observability_pipelines/packs/fortinet_firewall.md @@ -0,0 +1,19 @@ +--- +title: Fortinet Firewall +description: Learn more about the Fortinet Firewall pack. +--- + +## Overview + +Fortinet firewall logs record allowed, denied, and network traffic. + +What this pack does: + +- Drops DNS, health, and internal flows +- Normalizes fields, adds protocol context +- Keeps threats, denied, and anomalous logs + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/haproxy_ingress.md b/content/en/observability_pipelines/packs/haproxy_ingress.md new file mode 100644 index 0000000000000..587a6695589b3 --- /dev/null +++ b/content/en/observability_pipelines/packs/haproxy_ingress.md @@ -0,0 +1,19 @@ +--- +title: HAProxy Ingress +description: Learn more about the HAProxy Ingress pack. +--- + +## Overview + +HAProxy Ingress logs record how Kubernetes ingress traffic is routed and served. + +What this pack does: + +- Extracts key fields for analysis +- Drops routine health check and metrics endpoints +- Generates metrics to support cluster monitoring + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/istio_proxy.md b/content/en/observability_pipelines/packs/istio_proxy.md new file mode 100644 index 0000000000000..549fead42d67a --- /dev/null +++ b/content/en/observability_pipelines/packs/istio_proxy.md @@ -0,0 +1,19 @@ +--- +title: Istio Proxy +description: Learn more about the Istio Proxy pack. +--- + +## Overview + +Istio Proxy logs capture inbound and outbound traffic handled by Envoy. + +What this pack does: + +- Generates key HTTP metrics for latency, errors, and traffic +- Samples routine successful requests +- Drops low-value noise (health checks, static assets, empty `200`s) + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/netskope.md b/content/en/observability_pipelines/packs/netskope.md new file mode 100644 index 0000000000000..85d6a99113a10 --- /dev/null +++ b/content/en/observability_pipelines/packs/netskope.md @@ -0,0 +1,19 @@ +--- +title: Netskope +description: Learn more about the Netskope pack. +--- + +## Overview + +Netskope logs capture cloud app use, policies, and security events. + +What this pack does: + +- Samples high-volume categories like streaming +- Keeps DLP, malware, policy, and blocked events +- Drops routine SaaS and collaboration traffic + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/nginx.md b/content/en/observability_pipelines/packs/nginx.md new file mode 100644 index 0000000000000..578213b96bdd7 --- /dev/null +++ b/content/en/observability_pipelines/packs/nginx.md @@ -0,0 +1,19 @@ +--- +title: NGINX +description: Learn more about the NGINX pack. +--- + +## Overview + +NGINX logs record client requests, responses, and errors from the web server. + +What this pack does: + +- Exports request and error metrics +- Drops health checks, static assets, and internal traffic +- Parses logs and standardize key fields + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/okta.md b/content/en/observability_pipelines/packs/okta.md new file mode 100644 index 0000000000000..d52d420c52779 --- /dev/null +++ b/content/en/observability_pipelines/packs/okta.md @@ -0,0 +1,19 @@ +--- +title: Okta +description: Learn more about the Okta pack. +--- + +## Overview + +Okta logs show authentication, user activity, and policy events. + +What this pack does: + +- Samples routine logins and profile updates +- Keeps high-value security events +- Drops low-value or routine system events + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/palo_alto_firewall.md b/content/en/observability_pipelines/packs/palo_alto_firewall.md new file mode 100644 index 0000000000000..b5770fadd14a2 --- /dev/null +++ b/content/en/observability_pipelines/packs/palo_alto_firewall.md @@ -0,0 +1,19 @@ +--- +title: Palo Alto Firewall +description: Learn more about the Palo Alto Firewall pack. +--- + +## Overview + +Palo Alto Firewall logs capture traffic, threat, and system events. + +What this pack does: + +- Keeps detections and enforcement logs +- Drops redundant and benign events +- Normalizes traffic, threat, and system fields + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/windows_xml.md b/content/en/observability_pipelines/packs/windows_xml.md new file mode 100644 index 0000000000000..522ba769c75b8 --- /dev/null +++ b/content/en/observability_pipelines/packs/windows_xml.md @@ -0,0 +1,19 @@ +--- +title: Windows XML +description: Learn more about the Windows XML pack. +--- + +## Overview + +Windows XML logs capture system, application, and security activity from Windows hosts. + +What this pack does: + +- Converts verbose XML into JSON +- Drops empty and redundant fields +- Keeps only security-relevant events + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/zscaler_zia_dns.md b/content/en/observability_pipelines/packs/zscaler_zia_dns.md new file mode 100644 index 0000000000000..1e20bd44fadef --- /dev/null +++ b/content/en/observability_pipelines/packs/zscaler_zia_dns.md @@ -0,0 +1,19 @@ +--- +title: ZScaler ZIA DNS +description: Learn more about the ZScaler ZIA DNS pack. +--- + +## Overview + +ZScaler ZIA DNS logs capture org-wide DNS activity and policy actions. + +What this pack does: + +- Retains denied and unusual DNS lookups +- Samples high-volume allowed requests +- Filters out predictable internal traffic + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/zscaler_zia_firewall.md b/content/en/observability_pipelines/packs/zscaler_zia_firewall.md new file mode 100644 index 0000000000000..bbd516bad0479 --- /dev/null +++ b/content/en/observability_pipelines/packs/zscaler_zia_firewall.md @@ -0,0 +1,19 @@ +--- +title: Zscaler ZIA Firewall +description: Learn more about the Zscaler ZIA Firewall pack. +--- + +## Overview + +Zscaler ZIA Firewall logs show network traffic and security events. + +What this pack does: + +- Samples routine allowed and internal traffic +- Keeps blocked, threat, and unusual events +- Generates traffic and firewall metrics + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/zscaler_zia_tunnel.md b/content/en/observability_pipelines/packs/zscaler_zia_tunnel.md new file mode 100644 index 0000000000000..a3e27d98eede6 --- /dev/null +++ b/content/en/observability_pipelines/packs/zscaler_zia_tunnel.md @@ -0,0 +1,19 @@ +--- +title: Zscaler ZIA Tunnel +description: Learn more about the Zscaler ZIA Tunnel pack. +--- + +## Overview + +Zscaler ZIA Tunnel logs show tunnel health, traffic, and key events. + +What this pack does: + +- Reduces log volume of routine heartbeat logs +- Consolidates repeated failure alerts +- Standardizes fields for metrics and alerts + +## Example logs + +TKTK + diff --git a/content/en/observability_pipelines/packs/zscaler_zia_web_logs.md b/content/en/observability_pipelines/packs/zscaler_zia_web_logs.md new file mode 100644 index 0000000000000..c0095bcbea6bc --- /dev/null +++ b/content/en/observability_pipelines/packs/zscaler_zia_web_logs.md @@ -0,0 +1,19 @@ +--- +title: Zscaler ZIA Web Logs +description: Learn more about the Zscaler ZIA Web Logs pack. +--- + +## Overview + +Zscaler ZIA Web Logs capture user web activity and security actions. + +What this pack does: + +- Normalizes fields, adds threat context +- Drops routine and low-signal traffic +- Keeps risky, blocked, unknown traffic + +## Example logs + +TKTK + From 65cb7b6089967c41ce07c305bd0041fd91bfa203 Mon Sep 17 00:00:00 2001 From: May Lee Date: Fri, 31 Oct 2025 11:00:54 -0400 Subject: [PATCH 2/3] add individual packs --- config/_default/menus/main.en.yaml | 155 +++++++++++++++--- .../observability_pipelines/packs/_index.md | 62 ++++--- .../packs/amazon_cloudfront.md | 4 +- 3 files changed, 173 insertions(+), 48 deletions(-) diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index f20cc9011e31c..5b91ae1386b49 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -5599,121 +5599,226 @@ menu: parent: observability_pipelines_processors identifier: observability_pipelines_processors_throttle weight: 319 + - name: Packs + url: observability_pipelines/packs/ + parent: observability_pipelines + identifier: observability_pipelines_packs + weight: 4 + - name: Akamai CDN + url: observability_pipelines/packs/akamai_cdn/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_akamai_cdn + weight: 401 + - name: AWS CloudFront + url: observability_pipelines/packs/amazon_cloudfront/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_amazon_cloudfront + weight: 402 + - name: AWS CloudTrail + url: observability_pipelines/packs/aws_cloudtrail/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_aws_cloudtrail + weight: 403 + - name: Amazon VPC Flow Logs + url: observability_pipelines/packs/amazon_vpc_flow_logs/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_amazon_vpc_flow_logs + weight: 404 + - name: Cisco ASA + url: observability_pipelines/packs/cisco_asa/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_cisco_asa + weight: 405 + - name: Cloudflare + url: observability_pipelines/packs/cloudflare/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_cloudflare + weight: 406 + - name: F5 + url: observability_pipelines/packs/f5/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_f5 + weight: 407 + - name: Fastly + url: observability_pipelines/packs/fastly/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_fastly + weight: 408 + - name: Fortinet Firewall + url: observability_pipelines/packs/fortinet_firewall/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_fortinet_firewall + weight: 409 + - name: HAProxy Ingress + url: observability_pipelines/packs/haproxy_ingress/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_haproxy_ingress + weight: 410 + - name: Istio Proxy + url: observability_pipelines/packs/istio_proxy/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_istio_proxy + weight: 411 + - name: Netskope + url: observability_pipelines/packs/netskope/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_netskope + weight: 412 + - name: NGINX + url: observability_pipelines/packs/nginx/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_nginx + weight: 413 + - name: Okta + url: observability_pipelines/packs/okta/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_okta + weight: 414 + - name: Palo Alto Firewall + url: observability_pipelines/packs/palo_alto_firewall/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_palo_alto_firewall + weight: 415 + - name: Windows XML + url: observability_pipelines/packs/windows_xml/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_windows_xml + weight: 416 + - name: ZScaler ZIA DNS + url: observability_pipelines/packs/zscaler_zia_dns/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_zscaler_zia_dns + weight: 417 + - name: Zscaler ZIA Firewall + url: observability_pipelines/packs/zscaler_zia_firewall/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_zscaler_zia_firewall + weight: 418 + - name: Zscaler ZIA Tunnel + url: observability_pipelines/packs/zscaler_zia_tunnel/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_zscaler_zia_tunnel + weight: 419 + - name: Zscaler ZIA Web Logs + url: observability_pipelines/packs/zscaler_zia_web_logs/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_zscaler_zia_web_logs + weight: 420 - name: Destinations url: observability_pipelines/destinations/ parent: observability_pipelines identifier: observability_pipelines_destinations - weight: 4 + weight: 5 - name: Amazon OpenSearch url: observability_pipelines/destinations/amazon_opensearch/ parent: observability_pipelines_destinations identifier: observability_pipelines_amazon_opensearch - weight: 401 + weight: 501 - name: Amazon S3 identifier: observability_pipelines_destinations_amazon_s3 url: observability_pipelines/destinations/amazon_s3/ parent: observability_pipelines_destinations - weight: 402 + weight: 502 - name: Amazon Security Lake identifier: observability_pipelines_destinations_amazon_security_lake url: observability_pipelines/destinations/amazon_security_lake/ parent: observability_pipelines_destinations - weight: 403 + weight: 503 - name: Azure Storage identifier: observability_pipelines_azure_storage url: observability_pipelines/destinations/azure_storage/ parent: observability_pipelines_destinations - weight: 404 + weight: 504 - name: CrowdStrike NG-SIEM identifier: observability_pipelines_crowdstrike_ng_siem url: observability_pipelines/destinations/crowdstrike_ng_siem/ parent: observability_pipelines_destinations - weight: 405 + weight: 505 - name: Datadog CloudPrem url: observability_pipelines/destinations/cloudprem/ parent: observability_pipelines_destinations identifier: observability_pipelines_datadog_cloudprem - weight: 406 + weight: 506 - name: Datadog Logs url: observability_pipelines/destinations/datadog_logs/ parent: observability_pipelines_destinations identifier: observability_pipelines_datadog_logs - weight: 407 + weight: 507 - name: Elasticsearch url: observability_pipelines/destinations/elasticsearch/ parent: observability_pipelines_destinations identifier: observability_pipelines_elasticsearch - weight: 408 + weight: 508 - name: Google Chronicle url: observability_pipelines/destinations/google_chronicle parent: observability_pipelines_destinations identifier: observability_pipelines_google_chronicle - weight: 409 + weight: 509 - name: Google Cloud Storage identifier: observability_pipelines_google_cloud_storage url: /observability_pipelines/destinations/google_cloud_storage/ parent: observability_pipelines_destinations - weight: 410 + weight: 510 - name: Google Pub/Sub identifier: observability_pipelines_google_pubsub url: /observability_pipelines/destinations/google_pubsub/ parent: observability_pipelines_destinations - weight: 411 + weight: 511 - name: HTTP Client url: observability_pipelines/destinations/http_client/ parent: observability_pipelines_destinations identifier: observability_pipelines_http_client - weight: 412 + weight: 512 - name: Kafka url: observability_pipelines/destinations/kafka/ parent: observability_pipelines_destinations identifier: observability_pipelines_kafka - weight: 413 + weight: 513 - name: Microsoft Sentinel identifier: observability_pipelines_microsoft_sentinel url: /observability_pipelines/destinations/microsoft_sentinel/ parent: observability_pipelines_destinations - weight: 414 + weight: 514 - name: New Relic identifier: observability_pipelines_new_relic url: /observability_pipelines/destinations/new_relic/ parent: observability_pipelines_destinations - weight: 415 + weight: 515 - name: OpenSearch url: observability_pipelines/destinations/opensearch parent: observability_pipelines_destinations identifier: observability_pipelines_opensearch - weight: 416 + weight: 516 - name: SentinelOne url: observability_pipelines/destinations/sentinelone parent: observability_pipelines_destinations identifier: observability_pipelines_sentinelone - weight: 417 + weight: 517 - name: Socket url: observability_pipelines/destinations/socket parent: observability_pipelines_destinations identifier: observability_pipelines_socket - weight: 418 + weight: 518 - name: Splunk HEC url: observability_pipelines/destinations/splunk_hec parent: observability_pipelines_destinations identifier: observability_pipelines_splunk_hec - weight: 419 + weight: 519 - name: Sumo Logic Hosted Collector url: observability_pipelines/destinations/sumo_logic_hosted_collector parent: observability_pipelines_destinations identifier: observability_pipelines_sumo_logic_hosted_collector - weight: 420 + weight: 520 - name: Syslog url: observability_pipelines/destinations/syslog parent: observability_pipelines_destinations identifier: observability_pipelines_syslog - weight: 421 + weight: 521 - name: Scaling and Performance url: observability_pipelines/scaling_and_performance/ parent: observability_pipelines identifier: observability_pipelines_scaling_and_performance - weight: 5 + weight: 6 - name: Handling Load and Backpressure url: observability_pipelines/scaling_and_performance/handling_load_and_backpressure/ parent: observability_pipelines_scaling_and_performance @@ -5728,7 +5833,7 @@ menu: url: observability_pipelines/monitoring_and_troubleshooting/ parent: observability_pipelines identifier: observability_pipelines_monitoring_and_troubleshooting - weight: 6 + weight: 7 - name: Worker CLI Commands url: observability_pipelines/monitoring_and_troubleshooting/worker_cli_commands/ parent: observability_pipelines_monitoring_and_troubleshooting @@ -5753,7 +5858,7 @@ menu: url: observability_pipelines/guide/ parent: observability_pipelines identifier: observability_pipelines_guide - weight: 7 + weight: 8 - name: Log Management url: logs/ pre: log diff --git a/content/en/observability_pipelines/packs/_index.md b/content/en/observability_pipelines/packs/_index.md index e2a2b32be978a..50de7eca23c60 100644 --- a/content/en/observability_pipelines/packs/_index.md +++ b/content/en/observability_pipelines/packs/_index.md @@ -44,27 +44,47 @@ To set up packs: These are the available packs: -- Akamai CDN -- AWS CloudFront -- AWS CloudTrail -- Amazon VPC Flow Logs -- Cisco ASA -- Cloudflare -- F5 -- Fastly -- Fortinet Firewall -- HAProxy Ingress -- Istio Proxy -- Netskope -- NGINX -- Okta -- Palo Alto Firewall -- Windows XML -- ZScaler ZIA DNS -- Zscaler ZIA Firewall -- Zscaler ZIA Tunnel -- Zscaler ZIA Web Logs +- [Akamai CDN][4] +- [AWS CloudFront][5] +- [AWS CloudTrail][6] +- [Amazon VPC Flow Logs][7] +- [Cisco ASA][8] +- [Cloudflare][9] +- [F5][10] +- [Fastly][11] +- [Fortinet Firewall][12] +- [HAProxy Ingress][13] +- [Istio Proxy][14] +- [Netskope][15] +- [NGINX][16] +- [Okta][17] +- [Palo Alto Firewall][18] +- [Windows XML][19] +- [ZScaler ZIA DNS][20] +- [Zscaler ZIA Firewall][21] +- [Zscaler ZIA Tunnel][22] +- [Zscaler ZIA Web Logs][23] [1]: https://app.datadoghq.com/observability-pipelines [2]: /observability_pipelines/processors/ -[3]: /observability_pipelines/set_up_pipelines/ \ No newline at end of file +[3]: /observability_pipelines/set_up_pipelines/ +[4]: /observability_pipelines/packs/akamai_cdn/ +[5]: /observability_pipelines/packs/amazon_cloudfront/ +[6]: /observability_pipelines/packs/aws_cloudtrail/ +[7]: /observability_pipelines/packs/amazon_vpc_flow_logs/ +[8]: /observability_pipelines/packs/cisco_asa/ +[9]: /observability_pipelines/packs/cloudflare/ +[10]: /observability_pipelines/packs/f5/ +[11]: /observability_pipelines/packs/fastly/ +[12]: /observability_pipelines/packs/fortinet_firewall/ +[13]: /observability_pipelines/packs/haproxy_ingress/ +[14]: /observability_pipelines/packs/istio_proxy/ +[15]: /observability_pipelines/packs/netskope/ +[16]: /observability_pipelines/packs/nginx/ +[17]: /observability_pipelines/packs/okta/ +[18]: /observability_pipelines/packs/palo_alto_firewall/ +[19]: /observability_pipelines/packs/windows_xml/ +[20]: /observability_pipelines/packs/zscaler_zia_dns/ +[21]: /observability_pipelines/packs/zscaler_zia_firewall/ +[22]: /observability_pipelines/packs/zscaler_zia_tunnel/ +[23]: /observability_pipelines/packs/zscaler_zia_web_logs/ \ No newline at end of file diff --git a/content/en/observability_pipelines/packs/amazon_cloudfront.md b/content/en/observability_pipelines/packs/amazon_cloudfront.md index 685b6282e0696..69a081396c765 100644 --- a/content/en/observability_pipelines/packs/amazon_cloudfront.md +++ b/content/en/observability_pipelines/packs/amazon_cloudfront.md @@ -1,11 +1,11 @@ --- -title: AWS CloudFront +title: Amazon CloudFront description: Learn more about the AWS CloudFront pack. --- ## Overview -AWS CloudFront logs show requests, cache use, and edge activity +AWS CloudFront logs show requests, cache use, and edge activity. What this pack does: From 16874dd93fd06f71ff9cd17185c7d3b9605ddd91 Mon Sep 17 00:00:00 2001 From: May Lee Date: Fri, 31 Oct 2025 13:34:19 -0400 Subject: [PATCH 3/3] edits --- config/_default/menus/main.en.yaml | 12 ++--- .../observability_pipelines/packs/_index.md | 50 +++++++++---------- .../observability_pipelines/packs/fastly.md | 2 +- .../packs/fortinet_firewall.md | 2 +- .../packs/istio_proxy.md | 2 +- .../packs/palo_alto_firewall.md | 2 +- .../packs/windows_xml.md | 2 +- .../packs/zscaler_zia_dns.md | 2 +- .../packs/zscaler_zia_firewall.md | 2 +- .../packs/zscaler_zia_tunnel.md | 2 +- .../packs/zscaler_zia_web_logs.md | 4 +- 11 files changed, 41 insertions(+), 41 deletions(-) diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 5b91ae1386b49..21bea466d2c1f 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -5609,20 +5609,20 @@ menu: parent: observability_pipelines_packs identifier: observability_pipelines_packs_akamai_cdn weight: 401 + - name: Amazon VPC Flow Logs + url: observability_pipelines/packs/amazon_vpc_flow_logs/ + parent: observability_pipelines_packs + identifier: observability_pipelines_packs_amazon_vpc_flow_logs + weight: 402 - name: AWS CloudFront url: observability_pipelines/packs/amazon_cloudfront/ parent: observability_pipelines_packs identifier: observability_pipelines_packs_amazon_cloudfront - weight: 402 + weight: 403 - name: AWS CloudTrail url: observability_pipelines/packs/aws_cloudtrail/ parent: observability_pipelines_packs identifier: observability_pipelines_packs_aws_cloudtrail - weight: 403 - - name: Amazon VPC Flow Logs - url: observability_pipelines/packs/amazon_vpc_flow_logs/ - parent: observability_pipelines_packs - identifier: observability_pipelines_packs_amazon_vpc_flow_logs weight: 404 - name: Cisco ASA url: observability_pipelines/packs/cisco_asa/ diff --git a/content/en/observability_pipelines/packs/_index.md b/content/en/observability_pipelines/packs/_index.md index 50de7eca23c60..28406fa1a6f90 100644 --- a/content/en/observability_pipelines/packs/_index.md +++ b/content/en/observability_pipelines/packs/_index.md @@ -6,22 +6,22 @@ disable_toc: false ## Overview -When you setup a pipeline to send a logs from a specific source to Observability Pipelines, you likely have to think about questions such as: +When you set up a pipeline to send logs from a specific source to Observability Pipelines, you might have questions such as: - Which logs from this source are important? -- Which logs from this source should I drop? -- Which logs should I retain? -- Should I sample logs? -- Should I add quotas? +- Which logs from this source should be dropped? +- Which logs should be retained? +- Should logs be sampled? +- Should quotas be added? -Oftentimes you also have to talk to different teams to find answers to these questions. +Oftentimes, you need to consult with different teams to answer these questions. -Observability Pipelines Packs are predefined configurations that help you set up and optimize Observability Pipelines without extensive manual configuration. Each pack is specific to a source and identifies: +Use Observability Pipelines Packs to help you set up and optimize Observability Pipelines without extensive manual configuration. Each pack contains predefined configurations that are specific to a source and identifies: -- Log fields that can safely be removed from the logs. -- Logs that can be dropped, such as duplicated logs. -- Logs that need to be parsed. -- Logs that need to be formatted for the destination. +- Log fields that can safely be removed +- Logs that can be dropped, such as duplicated logs +- Logs that need to be parsed +- Logs that need to be formatted for the destination ## Setup @@ -31,23 +31,23 @@ To set up packs: 1. Click **Packs**. 1. Click the pack you want to set up. 1. You can either create a new pipeline from the pack or add the pack to an existing pipelines. -- If you clicked **Add to New Pipeline**, the pack is added to a new pipeline. - - Click the processor group that was added to see the individual processors the pack added and edit them as needed. See [Processors][2] for more information. - - See [Set Up Pipelines][3] for information on setting up the pipeline. -- If you clicked **Add to Existing Pipeline**: - 1. Select the pipeline you want to add the pack to. - 1. Click **Add to Existing Pipeline**. - - The pack is added to the last processor group in your pipeline. - - Click on the group to review the individual processors and edit them as needed. See [Processors][2] for more information. + - If you clicked **Add to New Pipeline**, in the new pipeline that was created: + - Click the processor group that was added to see the individual processors that the pack added and edit them as needed. See [Processors][2] for more information. + - See [Set Up Pipelines][3] for information on setting up the rest of the pipeline. + - If you clicked **Add to Existing Pipeline**: + 1. Select the pipeline you want to add the pack to. + 1. Click **Add to Existing Pipeline**. + - The pack is added to the last processor group in your pipeline. + - Click on the group to review the individual processors and edit them as needed. See [Processors][2] for more information. ## Packs These are the available packs: - [Akamai CDN][4] -- [AWS CloudFront][5] -- [AWS CloudTrail][6] -- [Amazon VPC Flow Logs][7] +- [Amazon VPC Flow Logs][5] +- [AWS CloudFront][6] +- [AWS CloudTrail][7] - [Cisco ASA][8] - [Cloudflare][9] - [F5][10] @@ -69,9 +69,9 @@ These are the available packs: [2]: /observability_pipelines/processors/ [3]: /observability_pipelines/set_up_pipelines/ [4]: /observability_pipelines/packs/akamai_cdn/ -[5]: /observability_pipelines/packs/amazon_cloudfront/ -[6]: /observability_pipelines/packs/aws_cloudtrail/ -[7]: /observability_pipelines/packs/amazon_vpc_flow_logs/ +[5]: /observability_pipelines/packs/amazon_vpc_flow_logs/ +[6]: /observability_pipelines/packs/amazon_cloudfront/ +[7]: /observability_pipelines/packs/aws_cloudtrail/ [8]: /observability_pipelines/packs/cisco_asa/ [9]: /observability_pipelines/packs/cloudflare/ [10]: /observability_pipelines/packs/f5/ diff --git a/content/en/observability_pipelines/packs/fastly.md b/content/en/observability_pipelines/packs/fastly.md index 4121d14ce8491..b2e66e8768f7b 100644 --- a/content/en/observability_pipelines/packs/fastly.md +++ b/content/en/observability_pipelines/packs/fastly.md @@ -9,7 +9,7 @@ Fastly CDN logs record client requests, cache states, and delivery performance. What this pack does: -- Samples static assets and `200 OK`s +- Samples static assets and `200 OK` logs - Keeps cache errors and slow outliers - Normalizes fields, drops bulky headers diff --git a/content/en/observability_pipelines/packs/fortinet_firewall.md b/content/en/observability_pipelines/packs/fortinet_firewall.md index 45ebd457af004..67d3d1fa44ee3 100644 --- a/content/en/observability_pipelines/packs/fortinet_firewall.md +++ b/content/en/observability_pipelines/packs/fortinet_firewall.md @@ -5,7 +5,7 @@ description: Learn more about the Fortinet Firewall pack. ## Overview -Fortinet firewall logs record allowed, denied, and network traffic. +Fortinet firewall logs record allowed, denied, and other network traffic. What this pack does: diff --git a/content/en/observability_pipelines/packs/istio_proxy.md b/content/en/observability_pipelines/packs/istio_proxy.md index 549fead42d67a..78fedf4eb2717 100644 --- a/content/en/observability_pipelines/packs/istio_proxy.md +++ b/content/en/observability_pipelines/packs/istio_proxy.md @@ -11,7 +11,7 @@ What this pack does: - Generates key HTTP metrics for latency, errors, and traffic - Samples routine successful requests -- Drops low-value noise (health checks, static assets, empty `200`s) +- Drops low-value noise (health checks, static assets, empty `200` logs) ## Example logs diff --git a/content/en/observability_pipelines/packs/palo_alto_firewall.md b/content/en/observability_pipelines/packs/palo_alto_firewall.md index b5770fadd14a2..4006329ca257e 100644 --- a/content/en/observability_pipelines/packs/palo_alto_firewall.md +++ b/content/en/observability_pipelines/packs/palo_alto_firewall.md @@ -5,7 +5,7 @@ description: Learn more about the Palo Alto Firewall pack. ## Overview -Palo Alto Firewall logs capture traffic, threat, and system events. +Palo Alto firewall logs capture traffic, threat, and system events. What this pack does: diff --git a/content/en/observability_pipelines/packs/windows_xml.md b/content/en/observability_pipelines/packs/windows_xml.md index 522ba769c75b8..433b5f5becc89 100644 --- a/content/en/observability_pipelines/packs/windows_xml.md +++ b/content/en/observability_pipelines/packs/windows_xml.md @@ -5,7 +5,7 @@ description: Learn more about the Windows XML pack. ## Overview -Windows XML logs capture system, application, and security activity from Windows hosts. +Windows Event logs capture system, application, and security activity from Windows hosts. What this pack does: diff --git a/content/en/observability_pipelines/packs/zscaler_zia_dns.md b/content/en/observability_pipelines/packs/zscaler_zia_dns.md index 1e20bd44fadef..302d0dff71289 100644 --- a/content/en/observability_pipelines/packs/zscaler_zia_dns.md +++ b/content/en/observability_pipelines/packs/zscaler_zia_dns.md @@ -5,7 +5,7 @@ description: Learn more about the ZScaler ZIA DNS pack. ## Overview -ZScaler ZIA DNS logs capture org-wide DNS activity and policy actions. +ZScaler Internet Access (ZIA) DNS logs capture org-wide DNS activity and policy actions. What this pack does: diff --git a/content/en/observability_pipelines/packs/zscaler_zia_firewall.md b/content/en/observability_pipelines/packs/zscaler_zia_firewall.md index bbd516bad0479..3444b7e3a6604 100644 --- a/content/en/observability_pipelines/packs/zscaler_zia_firewall.md +++ b/content/en/observability_pipelines/packs/zscaler_zia_firewall.md @@ -5,7 +5,7 @@ description: Learn more about the Zscaler ZIA Firewall pack. ## Overview -Zscaler ZIA Firewall logs show network traffic and security events. +Zscaler Internet Access (ZIA) Firewall logs show network traffic and security events. What this pack does: diff --git a/content/en/observability_pipelines/packs/zscaler_zia_tunnel.md b/content/en/observability_pipelines/packs/zscaler_zia_tunnel.md index a3e27d98eede6..91c523b78207b 100644 --- a/content/en/observability_pipelines/packs/zscaler_zia_tunnel.md +++ b/content/en/observability_pipelines/packs/zscaler_zia_tunnel.md @@ -5,7 +5,7 @@ description: Learn more about the Zscaler ZIA Tunnel pack. ## Overview -Zscaler ZIA Tunnel logs show tunnel health, traffic, and key events. +Zscaler Internet Access (ZIA) Tunnel logs show tunnel health, traffic, and key events. What this pack does: diff --git a/content/en/observability_pipelines/packs/zscaler_zia_web_logs.md b/content/en/observability_pipelines/packs/zscaler_zia_web_logs.md index c0095bcbea6bc..92967977f8eed 100644 --- a/content/en/observability_pipelines/packs/zscaler_zia_web_logs.md +++ b/content/en/observability_pipelines/packs/zscaler_zia_web_logs.md @@ -5,11 +5,11 @@ description: Learn more about the Zscaler ZIA Web Logs pack. ## Overview -Zscaler ZIA Web Logs capture user web activity and security actions. +Zscaler Internet Access (ZIA) Web Logs capture user web activity and security actions. What this pack does: -- Normalizes fields, adds threat context +- Normalizes fields and adds threat context - Drops routine and low-signal traffic - Keeps risky, blocked, unknown traffic