-
Notifications
You must be signed in to change notification settings - Fork 44
Add AI Request Logging experiment with observability dashboard #149
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
Jameswlepage
wants to merge
10
commits into
WordPress:develop
Choose a base branch
from
Jameswlepage:feature/ai-request-logging
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
a43ffdf
Add AI Request Logging experiment
Jameswlepage 22384c7
Add Provider_Metadata_Registry dependency for log page
Jameswlepage f18326c
Add get_ai_icon_svg helper and icon asset
Jameswlepage 1ddc226
Initialize logging discovery strategy in experiment register
Jameswlepage b9285c7
Refactor logging to use SDK public API instead of reflection
Jameswlepage 3879d65
Add Log_Data_Extractor for extensible request parsing
Jameswlepage e3403b4
Merge branch 'develop' into feature/ai-request-logging
jeffpaul 864d452
basic commit to try and trigger playground generation
jeffpaul 498eece
Merge branch 'develop' into feature/ai-request-logging
dkotter 3731f8f
Add required NPM packages
dkotter File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| # AI Request Logging | ||
|
|
||
| ## Summary | ||
| Provides an opt-in observability surface that records every AI request (provider, model, duration, token counts, status, cost estimate) and exposes them through a React-powered dashboard under `Settings → AI Request Logs`. When enabled, the SDK's HTTP transporter is wrapped with a logging decorator using the public `setHttpTransporter()` API. | ||
|
|
||
| ## Key Hooks & Entry Points | ||
| - `WordPress\AI\Experiments\AI_Request_Logging\AI_Request_Logging::register()`: | ||
| - `rest_api_init` → registers `WordPress\AI\Logging\REST\AI_Request_Log_Controller`, which exposes `/ai/v1/logs`, `/summary`, `/filters`, and per-log endpoints. | ||
| - `is_admin()` guard → instantiates `WordPress\AI\Logging\AI_Request_Log_Page` to add the Settings submenu and enqueue assets. | ||
| - Plugin bootstrap initializes `WordPress\AI\Logging\Logging_Integration` (and `AI_Request_Log_Manager::init()`) when the experiment toggle is enabled, ensuring the transporter wrapper and daily cleanup job stay disabled otherwise. | ||
| - Database + cleanup are handled inside `AI_Request_Log_Manager::init()` (table creation, cron scheduling, option storage). | ||
|
|
||
| ## Architecture | ||
| The logging system uses the decorator pattern to wrap the SDK's HTTP transporter: | ||
|
|
||
| 1. `Logging_Integration::init()` is called during bootstrap when the experiment is enabled. | ||
| 2. On `wp_loaded` or `admin_init`, `Logging_Integration::wrap_transporter()`: | ||
| - Gets the current transporter from `AiClient::defaultRegistry()` | ||
| - Creates a `Logging_Http_Transporter` decorator around it | ||
| - Uses `registry->setHttpTransporter()` to swap in the logging version | ||
| 3. All subsequent AI requests go through the logging transporter, which records metrics before delegating to the underlying transporter. | ||
| 4. Data extraction is handled by `Log_Data_Extractor`, which parses request/response payloads and extracts provider, model, tokens, and previews. | ||
|
|
||
| This approach uses the SDK's public API rather than reflection or internal hacks, making it resilient to SDK updates. | ||
|
|
||
| ## Filter Hooks | ||
| The logging system exposes several filter hooks for extensibility: | ||
|
|
||
| ### `ai_request_log_providers` | ||
| Filters the provider detection patterns. Allows adding custom providers or modifying detection patterns. | ||
|
|
||
| ```php | ||
| add_filter( 'ai_request_log_providers', function( $patterns ) { | ||
| $patterns['my_provider'] = array( 'my-api.com', 'api.myprovider.io' ); | ||
| return $patterns; | ||
| } ); | ||
| ``` | ||
|
|
||
| ### `ai_request_log_context` | ||
| Filters the log context data before it's stored. Allows adding custom context or removing sensitive data. | ||
|
|
||
| ```php | ||
| add_filter( 'ai_request_log_context', function( $context, $decoded, $log_data ) { | ||
| $context['custom_field'] = 'custom_value'; | ||
| unset( $context['sensitive_field'] ); | ||
| return $context; | ||
| }, 10, 3 ); | ||
| ``` | ||
|
|
||
| ### `ai_request_log_tokens` | ||
| Filters the extracted token usage. Allows custom providers to supply their own token extraction logic. | ||
|
|
||
| ```php | ||
| add_filter( 'ai_request_log_tokens', function( $tokens, $response ) { | ||
| if ( isset( $response['my_token_field'] ) ) { | ||
| $tokens['input'] = $response['my_token_field']['in']; | ||
| $tokens['output'] = $response['my_token_field']['out']; | ||
| } | ||
| return $tokens; | ||
| }, 10, 2 ); | ||
| ``` | ||
|
|
||
| ### `ai_request_log_kind` | ||
| Filters the detected request kind (text, image, embeddings, audio). | ||
|
|
||
| ```php | ||
| add_filter( 'ai_request_log_kind', function( $kind, $provider, $path, $payload ) { | ||
| if ( str_contains( $path, '/my-custom-endpoint' ) ) { | ||
| return 'custom'; | ||
| } | ||
| return $kind; | ||
| }, 10, 4 ); | ||
| ``` | ||
|
|
||
| ## Assets & Data Flow | ||
| 1. When `AI Request Logs` is visited, `Asset_Loader` enqueues `admin/ai-request-logs` (`src/admin/ai-request-logs/index.tsx`) plus its stylesheet. The localized payload (`window.AiRequestLogsSettings`) includes REST routes, a nonce, and initial state (enabled flag, retention days, summary, filters). | ||
| 2. The React app: | ||
| - Configures `@wordpress/api-fetch` with the nonce/root. | ||
| - Fetches logs (`GET /ai/v1/logs` with search/filter params) and displays them in a table with pagination. | ||
| - Fetches summaries (`GET /ai/v1/logs/summary`) for the KPI cards and filter metadata (`GET /ai/v1/logs/filters`). | ||
| - Posts to `/ai/v1/logs` to toggle logging and retention, and sends `DELETE /ai/v1/logs` to purge the table. | ||
| 3. On the backend, every AI HTTP request flows through `Logging_Http_Transporter`, which records metrics via `AI_Request_Log_Manager::log()` before returning the response to callers. Logs are stored in the `wp_ai_request_logs` table alongside JSON context for later inspection. | ||
|
|
||
| ## Testing | ||
| 1. Enable Experiments globally, toggle **AI Request Logging**, and ensure valid AI credentials exist (the experiment won't enable otherwise). | ||
| 2. Trigger an AI-powered feature (e.g., Type Ahead or Title Generation) so the system issues at least one completion request. | ||
| 3. Navigate to `Settings → AI Request Logs`. Confirm the chart and table populate and that the "Logging enabled" toggle reflects the settings page switch. | ||
| 4. Change the retention days value, save, and verify the option persists (reload the page or inspect `ai_request_logs_retention_days`). | ||
| 5. Click "Purge logs", confirm the success notice, and check the table empties. | ||
| 6. Disable the experiment and reload a front-end AI feature; no new rows should appear, and the logging integration should remain inactive. | ||
|
|
||
| ## Notes | ||
| - The HTTP logging layer only boots when both the global experiment switch and the `ai-request-logging` toggle are on, preventing unnecessary DB tables or cron events on installs that don't need observability. | ||
| - REST endpoints require `manage_options`. | ||
| - Model cost estimates rely on the static pricing table inside `AI_Request_Cost_Calculator`; update it as provider pricing evolves. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we use the colored variant like https://github.com/WordPress/ai/blob/develop/.wordpress-org/icon.svg instead?