Enhance GCP plugin with Cloud Logging API source and new request log table

[recharge-jmog]

Add Cloud Logging API Source and Request Log Table for Cloud Armor Logs

Overview

This pull request introduces two major enhancements to the GCP Tailpipe plugin:

1. Cloud Logging API Source

   • Adds a new source (cloud_logging_api) that enables direct collection of logs from the Google Cloud Logging API.
   • Supports configuration of log types and project selection, allowing flexible ingestion of various GCP log streams.

2. Request Log Table for Cloud Armor Logs

   • Introduces a new table (requests_log) designed to store and query Cloud Armor-augmented Application Load Balancer request logs.
   • The table schema captures key Cloud Armor fields, including security policy actions, threat intelligence, cache decisions, and detailed HTTP request/response metadata, as well as verbose logging fields.
   • Includes robust mapping logic to handle both API and bucket-based log ingestion, ensuring compatibility with different GCP log delivery methods.

Key Features

• New Source: cloud_logging_api

  • Collects logs directly from GCP Cloud Logging.
  • Configurable log type filtering (via source code modification, not in configuration file)
  • Integrates with existing connection and partition configuration patterns.

• New Table: requests_log

  • Supports analysis of Cloud Armor security events and HTTP(S) request flows.
  • Exposes fields for security policy enforcement, preview actions, threat intelligence, verbose logging, and more.

Motivation

These additions enable users to:

• Ingest and analyze Cloud Armor logs for security monitoring, threat detection, and compliance reporting.
• Query HTTP(S) request logs with full context on security policy actions and outcomes.
• Leverage Tailpipe’s SQL interface for advanced analytics on GCP network security events.

Documentation

• Example configuration and usage instructions are provided in the updated documentation.
• Table and source schemas are documented for reference.

---

CC: @turbot/tailpipe-team

[Copilot]

Pull Request Overview

This pull request enhances the GCP Tailpipe plugin by adding comprehensive Cloud Armor and application load balancer log support through two major additions:

• New cloud_logging_api source that enables direct collection of logs from the Google Cloud Logging API
• New requests_log table designed specifically for Cloud Armor-augmented Application Load Balancer request logs with detailed security policy analysis

Reviewed Changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tables/requests_log/requests_log_table.go	Implements the main table structure with source metadata configuration and row enrichment logic
tables/requests_log/requests_log_mapper.go	Provides mapping logic to transform both API and bucket-based log data into the RequestsLog structure
tables/requests_log/requests_log.go	Defines the core data structures and schema for Cloud Armor request logs
sources/cloud_logging_api/cloud_logging_api_source_config.go	Configuration validation for the new Cloud Logging API source
sources/cloud_logging_api/cloud_logging_api_source.go	Main source implementation for collecting logs directly from GCP Cloud Logging API
gcp/plugin.go	Plugin registration updates to include the new table and source
docs/tables/gcp_requests_log/index.md	Documentation for the new requests log table with usage examples
docs/sources/gcp_cloud_logging_api.md	Documentation for the new Cloud Logging API source

[ParthaI]

Hello @recharge-jmog,

Apologies for the delay in reviewing the PR — thank you for your patience! The changes you've made look great overall.

That said, I have a few thoughts on the approach. Introducing a separate source specifically for one log type might not be ideal, especially since the API call logic, input parameters, and code are essentially the same. Additionally, GCP supports multiple types of request logs, such as:

• appengine.googleapis.com%2Frequest_log
• run.googleapis.com%2Frequests
• projects/{projectID}/logs/requests (Load Balancer)

Creating a separate source for each log type could lead to redundancy and make maintenance harder.

I do agree, however, that the existing source name gcp_audit_log_api doesn’t fully reflect its broader capability. We’re considering renaming it to something like gcp_cloud_log_api to better align with its purpose — that is, supporting API calls for various log types, not just audit logs.

In the meantime, I’ve made a few updates:

• Adjusted the existing gcp_audit_log_api source to include filters for request logs like projects/{projectID}/logs/requests.

• Added a new table, gcp_requests_log, based on your PR.

• Introduced the new source gcp_cloud_log_api, which supports multiple log types (e.g., Cloud Run and App Engine). You can now collect these logs by specifying:

  log_types = ["cloud_run_request", "app_engine_request"]

  in the Tailpipe connection config.

All these changes are pushed to the branch: update-audit-log-api-source-to-add-request-logs

It would be great if you could test this out from the PR branch and share any feedback. I really appreciate your contribution and look forward to your thoughts!

Config:

partition "gcp_requests_log" "my_logs_requests" {
  source "gcp_audit_log_api" {
    connection = connection.gcp.cred
    log_types = ["requests"]
  }
}

Result:

 tpc gcp_requests_log.my_logs_requests --from T-1Y

Collecting logs for gcp_requests_log.my_logs_requests from 2024-08-12 to 2025-08-12

Rows:
  Received: 114
  Enriched: 114
  Saved:    114

Files:
  Compacted: 1 => 1

Completed: 31s


> select * from gcp_requests_log
+-------------------------------+---------------------------------------------------------+-------------------------------+-------------------------------------------------------------------------->
| backend_target_project_number | cache_decision                                          | enforced_edge_security_policy | enforced_security_policy                                                 >
+-------------------------------+---------------------------------------------------------+-------------------------------+-------------------------------------------------------------------------->
| projects/************         | [RESPONSE_HAS_CONTENT_TYPE CACHE_MODE_CACHE_ALL_STATIC] | <null>                        | map[configured_action:THROTTLE name:default-security-policy-for-backend-s>
| projects/************         | [RESPONSE_HAS_CONTENT_TYPE CACHE_MODE_CACHE_ALL_STATIC] | <null>                        | map[configured_action:THROTTLE name:default-security-policy-for-backend-s>
| projects/************         | [RESPONSE_HAS_CONTENT_TYPE CACHE_MODE_CACHE_ALL_STATIC] | <null>                        | map[configured_action:THROTTLE name:default-security-policy-for-backend-s>
| projects/************         | [RESPONSE_HAS_CONTENT_TYPE CACHE_MODE_CACHE_ALL_STATIC] | <null>                        | map[configured_action:THROTTLE name:default-security-policy-for-backend-s>
| projects/************         | [RESPONSE_HAS_CONTENT_TYPE CACHE_MODE_CACHE_ALL_STATIC] | <null>                        | map[configured_action:THROTTLE name:default-security-policy-for-backend-s>
| projects/************         | [RESPONSE_HAS_CONTENT_TYPE CACHE_MODE_CACHE_ALL_STATIC] | <null>                        | map[configured_action:THROTTLE name:default-security-policy-for-backend-s>

Thanks!

[recharge-jmog]

@ParthaI I'll give this a test and let you know. Sounds good in principle!

[recharge-jmog]

@ParthaI Ok, yeah aside from the known lack of rate limiting (future PR for me I guess, haha) on the API collection, it's working fine here. It wouldn't be that hard for someone to implement a new table based on the log formats available at https://cloud.google.com/logging/docs/api/platform-logs

Do you want to handle the merging with the main branch or should I incorporate your changes into my fork and open a new PR?

[ParthaI]

Hi @recharge-jmog, thank you for your feedback and for testing things out in the PR branch.

We are currently working on a design that will allow us to retrieve any type of logs. As part of this, we may also need to adjust source naming, config arguments, and related aspects. However, the design is not yet finalized, and we plan to prioritize this work this week.

Do you want to handle the merging with the main branch, or should I incorporate your changes into my fork and open a new PR?

The changes I pushed in the branch update-audit-log-api-source-to-add-request-logs are still at an early stage of the design updates. I will be making additional modifications to the audit_log_api source, so I think it may be best to hold off on making changes in this PR for now. Once we finalize the source design on our side, I’ll inform you so that you can update your changes accordingly.

Thanks again for your patience and understanding, and apologies for the inconvenience. 🙏

[recharge-jmog]

Hi @recharge-jmog, thank you for your feedback and for testing things out in the PR branch.
...
Thanks again for your patience and understanding, and apologies for the inconvenience. 🙏

Not a problem @ParthaI , just wanted to make sure I was aligned with what you're doing over there. Look forward to the final results, and let me know if you need some real-world testing of the final implementation. Thanks!

[ParthaI]

Hello @recharge-jmog,

Just a quick update — I’ve opened a PR that renames the audit_log_api source to logging_log_entry. This update is designed to support all types of log tables, ensuring broader compatibility. Additionally, I’ve added a comment
to the PR highlighting an overview of the changes introduced in this update.

I’ve reverted the request_log table changes in this commit
, so you should be able to rebase your PR once these changes are merged. Additionally, I’ve implemented a connection-level retry mechanism, which can be customized via the retry configuration parameters. I’ve also tested the request_log table and can confirm it’s working as expected.

Here’s a sample configuration for reference:

connection "gcp" "cred" {
  project            = "parker-aaa"
  credentials        = "/path/to/your/credential.json"
  min_retry_delay    = 100
  max_retry_delay    = 10000
  backoff_multiplier = 1.2
}

partition "gcp_audit_log" "my_logs_api" {
  source "gcp_logging_log_entry" {
    connection = connection.gcp.cred
    log_types  = ["data_access", "activity", "system_event", "policy"]
  }
}

With these changes, I was able to successfully retrieve logs spanning the past 6 months, pulling over 300k entries without encountering any rate-limiting errors.

It would be great if you could try this out locally by rebasing your PR with mine and sharing any feedback or observations.

The PR is currently under review, and we might refine some of the logic if necessary.

Thanks!

[github-actions]

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.