[codex] Add cost fields to LLM usage analytics#721
Merged
rahulkarajgikar merged 2 commits intotruffle-ai:mainfrom Apr 14, 2026
Merged
[codex] Add cost fields to LLM usage analytics#721rahulkarajgikar merged 2 commits intotruffle-ai:mainfrom
rahulkarajgikar merged 2 commits intotruffle-ai:mainfrom
Conversation
📝 WalkthroughWalkthroughThis PR extends LLM usage analytics across the dexto monorepo by adding cost breakdown fields to event types and propagating them through core events, server handlers, and CLI/WebUI analytics capture pipelines. USD cost estimation now includes per-category breakdowns (input, output, reasoning, cache operations) alongside total estimated costs. Changes
Sequence DiagramsequenceDiagram
participant LLM as LLM Provider
participant Core as Core Executor
participant Meta as Usage Metadata
participant Events as Event Emitters
participant Server as Server Handlers
participant Analytics as Analytics Capture
LLM->>Core: Response with tokens
Core->>Meta: Calculate cost breakdown
Meta->>Meta: calculateCostBreakdown()<br/>(per-category USD costs)
Core->>Events: emitLLMResponse(event, config)
Events->>Events: Include costBreakdown<br/>in llm:response payload
Events->>Server: llm:response event
Server->>Server: Extract costBreakdown<br/>from payload
Server->>Analytics: SSE/Usage events
Analytics->>Analytics: Map costBreakdown<br/>to USD fields
Analytics->>Analytics: Emit dexto_llm_tokens_consumed<br/>with cost data
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@rahulkarajgikar is attempting to deploy a commit to the Shaunak's projects Team on Vercel. A member of the Team first needs to authorize it. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (4)
packages/server/src/events/usage-event-subscriber.ts (1)
157-158: Extract cost-breakdown resolution into a dedicated resolver.Line 157 introduces a multi-source fallback chain inline. Please move this into a small resolver function to keep one explicit selection path and improve maintainability.
♻️ Suggested refactor
+ private resolveCostBreakdown( + payload: AgentEventMap['llm:response'] + ): AgentEventMap['llm:response']['costBreakdown'] | undefined { + if (payload.costBreakdown) { + return payload.costBreakdown; + } + if (!payload.provider || !payload.model || !payload.tokenUsage) { + return undefined; + } + const pricing = getModelPricing(payload.provider, payload.model); + return pricing ? calculateCostBreakdown(payload.tokenUsage, pricing) : undefined; + } + private buildUsageEvent(payload: AgentEventMap['llm:response']): UsageEvent | null { @@ - const resolvedCostBreakdown = - payload.costBreakdown ?? - (payload.provider && payload.model - ? (() => { - const pricing = getModelPricing(payload.provider, payload.model); - if (!pricing) { - return undefined; - } - - return calculateCostBreakdown(payload.tokenUsage, pricing); - })() - : undefined); + const resolvedCostBreakdown = this.resolveCostBreakdown(payload);As per coding guidelines: "Avoid multi-source values encoded as optional + fallback + fallback chains (a ?? b ?? c); prefer a single source of truth with explicit resolver function."
Also applies to: 167-167
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/server/src/events/usage-event-subscriber.ts` around lines 157 - 158, Extract the inline multi-source fallback used to compute costBreakdown into a dedicated resolver function (e.g., resolveCostBreakdown) and replace the inline expression (the payload.costBreakdown ?? (payload.provider && payload.model ... ) and the similar occurrence around line 167) with a single call to that resolver; the resolver should accept the payload (or provider, model, and any rawCost fields) and implement the explicit selection logic (check payload.costBreakdown first, then provider+model-derived value, etc.), returning the resolved costBreakdown value so the main flow in usage-event-subscriber.ts remains a single clear selection point.packages/webui/lib/events/handlers.ts (1)
242-272: Consider extracting the duplicatedcaptureTokenUsagecall into a helper.The same analytics capture logic appears twice in
handleLLMResponse(lines 242-272 for the streaming path and lines 320-350 for the non-streaming path). This internal duplication could lead to drift if one path is updated but not the other.♻️ Suggested extraction
+function captureTokenUsageAnalytics( + sessionId: string, + provider: string | undefined, + model: string | undefined, + tokenUsage: EventByName<'llm:response'>['tokenUsage'], + estimatedCost: number | undefined, + costBreakdown: EventByName<'llm:response'>['costBreakdown'], + estimatedInputTokens: number | undefined, + reasoningVariant: string | undefined, + reasoningBudgetTokens: number | undefined +): void { + if (!hasMeaningfulTokenUsageForAnalytics(tokenUsage, estimatedCost)) { + return; + } + + let estimateAccuracyPercent: number | undefined; + const actualInputTokens = tokenUsage?.inputTokens; + if (estimatedInputTokens !== undefined && actualInputTokens) { + const diff = estimatedInputTokens - actualInputTokens; + estimateAccuracyPercent = Math.round((diff / actualInputTokens) * 100); + } + + captureTokenUsage({ + sessionId, + provider, + model, + reasoningVariant, + reasoningBudgetTokens, + inputTokens: tokenUsage?.inputTokens, + outputTokens: tokenUsage?.outputTokens, + reasoningTokens: tokenUsage?.reasoningTokens, + totalTokens: tokenUsage?.totalTokens, + cacheReadTokens: tokenUsage?.cacheReadTokens, + cacheWriteTokens: tokenUsage?.cacheWriteTokens, + estimatedCostUsd: estimatedCost, + inputCostUsd: costBreakdown?.inputUsd, + outputCostUsd: costBreakdown?.outputUsd, + reasoningCostUsd: costBreakdown?.reasoningUsd, + cacheReadCostUsd: costBreakdown?.cacheReadUsd, + cacheWriteCostUsd: costBreakdown?.cacheWriteUsd, + estimatedInputTokens, + estimateAccuracyPercent, + }); +}Then call
captureTokenUsageAnalytics(sessionId, provider, model, tokenUsage, estimatedCost, costBreakdown, estimatedInputTokens, reasoningVariant, reasoningBudgetTokens)in both code paths.Also applies to: 320-350
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/webui/lib/events/handlers.ts` around lines 242 - 272, There is duplicated analytics logic in handleLLMResponse: the captureTokenUsage call (using sessionId, provider, model, reasoningVariant, reasoningBudgetTokens, tokenUsage, estimatedInputTokens, estimatedCost, costBreakdown, etc.) appears in both the streaming and non-streaming paths; extract this into a new helper function (e.g., captureTokenUsageAnalytics) that accepts the necessary symbols (sessionId, provider, model, tokenUsage, estimatedCost, costBreakdown, estimatedInputTokens, reasoningVariant, reasoningBudgetTokens) and performs the estimateAccuracyPercent calculation (using estimatedInputTokens and tokenUsage.inputTokens) and then calls captureTokenUsage with the consolidated payload; replace the duplicated captureTokenUsage blocks in both paths with a call to this new helper to ensure single-source logic.packages/tui/src/services/processStream.ts (1)
119-139: Code duplication:hasMeaningfulTokenUsageForAnalyticsis duplicated with WebUI.This helper is identical to the one in
packages/webui/lib/events/handlers.ts(lines 124-144). Consider extracting it to a shared location to avoid divergence.Options:
- Add to
@dexto/corealongsidehasMeaningfulTokenUsageinpackages/core/src/llm/usage-metadata.ts- Add to a shared analytics utilities module
This would also make the behavioral difference between
hasMeaningfulTokenUsage(token-only gating) andhasMeaningfulTokenUsageForAnalytics(cost OR token gating) explicit and documented in one place.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/tui/src/services/processStream.ts` around lines 119 - 139, Extract the duplicated hasMeaningfulTokenUsageForAnalytics helper into a shared location and import it where needed: create/export a function named hasMeaningfulTokenUsageForAnalytics (with the same signature and logic) alongside hasMeaningfulTokenUsage in the core usage-metadata module (e.g., add to the module that defines hasMeaningfulTokenUsage), then replace the local implementations in processStream.ts and WebUI handlers.ts with imports from that shared module; ensure the exported symbol is typed correctly (matching Extract<StreamingEvent, { name: 'llm:response' }>['tokenUsage']) and update any imports/usages accordingly.packages/webui/lib/events/handlers.test.ts (1)
202-245: Good test coverage for the happy path; consider adding edge case coverage.The test correctly validates that cost breakdown fields are mapped to the analytics payload. However, there are two edge cases worth covering:
- When
costBreakdownis undefined butestimatedCostis present (the new gating logic should still emit analytics)- When
tokenUsagehas zero tokens butestimatedCostis defined (verifies the newhasMeaningfulTokenUsageForAnalyticsbehavior change)These would ensure the behavioral change (emitting analytics when only
estimatedCostis present) is regression-tested.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/webui/lib/events/handlers.test.ts` around lines 202 - 245, Add two new unit tests in handlers.test.ts that call handleLLMResponse: (1) a case where event.costBreakdown is undefined but event.estimatedCost is set—set up the chat message via useChatStore.getState().setStreamingMessage as other tests do and assert captureTokenUsageMock is called with estimatedCostUsd equal to the event.estimatedCost; (2) a case where event.tokenUsage has zeros (inputTokens/outputTokens/totalTokens = 0) but event.estimatedCost is set—again assert captureTokenUsageMock is invoked and includes estimatedCostUsd, confirming hasMeaningfulTokenUsageForAnalytics no longer blocks analytics when estimatedCost is present; reuse TEST_SESSION_ID, provider/model fields and the same expect.objectContaining pattern used in the existing test to verify the mapped cost fields.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@packages/server/src/events/usage-event-subscriber.ts`:
- Around line 157-158: Extract the inline multi-source fallback used to compute
costBreakdown into a dedicated resolver function (e.g., resolveCostBreakdown)
and replace the inline expression (the payload.costBreakdown ??
(payload.provider && payload.model ... ) and the similar occurrence around line
167) with a single call to that resolver; the resolver should accept the payload
(or provider, model, and any rawCost fields) and implement the explicit
selection logic (check payload.costBreakdown first, then provider+model-derived
value, etc.), returning the resolved costBreakdown value so the main flow in
usage-event-subscriber.ts remains a single clear selection point.
In `@packages/tui/src/services/processStream.ts`:
- Around line 119-139: Extract the duplicated
hasMeaningfulTokenUsageForAnalytics helper into a shared location and import it
where needed: create/export a function named hasMeaningfulTokenUsageForAnalytics
(with the same signature and logic) alongside hasMeaningfulTokenUsage in the
core usage-metadata module (e.g., add to the module that defines
hasMeaningfulTokenUsage), then replace the local implementations in
processStream.ts and WebUI handlers.ts with imports from that shared module;
ensure the exported symbol is typed correctly (matching Extract<StreamingEvent,
{ name: 'llm:response' }>['tokenUsage']) and update any imports/usages
accordingly.
In `@packages/webui/lib/events/handlers.test.ts`:
- Around line 202-245: Add two new unit tests in handlers.test.ts that call
handleLLMResponse: (1) a case where event.costBreakdown is undefined but
event.estimatedCost is set—set up the chat message via
useChatStore.getState().setStreamingMessage as other tests do and assert
captureTokenUsageMock is called with estimatedCostUsd equal to the
event.estimatedCost; (2) a case where event.tokenUsage has zeros
(inputTokens/outputTokens/totalTokens = 0) but event.estimatedCost is set—again
assert captureTokenUsageMock is invoked and includes estimatedCostUsd,
confirming hasMeaningfulTokenUsageForAnalytics no longer blocks analytics when
estimatedCost is present; reuse TEST_SESSION_ID, provider/model fields and the
same expect.objectContaining pattern used in the existing test to verify the
mapped cost fields.
In `@packages/webui/lib/events/handlers.ts`:
- Around line 242-272: There is duplicated analytics logic in handleLLMResponse:
the captureTokenUsage call (using sessionId, provider, model, reasoningVariant,
reasoningBudgetTokens, tokenUsage, estimatedInputTokens, estimatedCost,
costBreakdown, etc.) appears in both the streaming and non-streaming paths;
extract this into a new helper function (e.g., captureTokenUsageAnalytics) that
accepts the necessary symbols (sessionId, provider, model, tokenUsage,
estimatedCost, costBreakdown, estimatedInputTokens, reasoningVariant,
reasoningBudgetTokens) and performs the estimateAccuracyPercent calculation
(using estimatedInputTokens and tokenUsage.inputTokens) and then calls
captureTokenUsage with the consolidated payload; replace the duplicated
captureTokenUsage blocks in both paths with a call to this new helper to ensure
single-source logic.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: f7c5771b-16a5-4b02-8359-539f76ab3c3d
📒 Files selected for processing (13)
.changeset/few-apes-judge.mddocs/static/openapi/openapi.jsonpackages/analytics/src/events.tspackages/core/src/events/index.tspackages/core/src/llm/executor/stream-processor.test.tspackages/core/src/llm/executor/stream-processor.tspackages/core/src/llm/usage-metadata.tspackages/server/src/events/a2a-sse-subscriber.tspackages/server/src/events/usage-event-subscriber.tspackages/tui/src/services/processStream.test.tspackages/tui/src/services/processStream.tspackages/webui/lib/events/handlers.test.tspackages/webui/lib/events/handlers.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dexto_llm_tokens_consumedanalytics payloadcostBreakdownthrough corellm:responsepricing metadata so TUI/WebUI analytics can forward cost data without duplicating pricing logicWhy
Core already computes per-response LLM pricing, but the cross-platform usage analytics event only carried token counts. That left PostHog-style usage metrics blind to cost even though the information already existed in the runtime.
Impact
inputTokens/outputTokensare zeroValidation
pnpm exec vitest run packages/core/src/llm/executor/stream-processor.test.ts packages/webui/lib/events/handlers.test.ts packages/tui/src/services/processStream.test.tsbash scripts/quality-checks.sh lintbash scripts/quality-checks.sh typecheckbash scripts/quality-checks.sh hono-inference./scripts/quality-checks.shreaches build and OpenAPI successfully, but the full test step is currently blocked by unrelated CLI test instability/timeouts inpackages/cli/src/cli/utils/config-validation.test.tsandpackages/cli/src/cli/modes/cli.test.tspnpm exec vitest run packages/cli/src/cli/modes/cli.test.ts packages/cli/src/cli/utils/config-validation.test.tspassed when rerun in isolationSummary by CodeRabbit