LLM Token Tracking

[cooper-grc]

Summary by cubic

Adds end-to-end LLM token usage tracking for agent runs, from parsing stream output to persisting normalized totals for billing and analytics. Also adds updated_after and updated_before filters to ListRuns for easier querying by last change time.

• New Features
  • Workers parse Claude stream output across interactive, sandbox, and PTY runs, aggregate per-turn usage, and attach it to RunExecutionResult.
  • worker.proto and Gateway client report usage (input, output, cache create/read, total) with SetTaskResult; WorkerService includes it in outbox payloads.
  • Orchestration extracts usage, adds it to completion snapshots, normalizes totals, and persists deltas in agent_run.usage_json (versioned, with billing_total_tokens); DLQ entries include usage details.
  • API: ListRuns supports updated_after and updated_before filters.

^{Written for commit 84acd44. Summary will update on new commits.}

[github-actions]

✅ Integration Tests PASSED

Gateway: https://api.airstore.ai
Query path: /sources/gmail/unread-emails
Timestamp: 2026-03-06T17:38:07Z

I/O Smoke Tests

Test	Status	Latency	Bytes
list_sources	PASS	360ms	-
list_gmail	PASS	405ms	-
list_query_path	PASS	863ms	-
read_file	PASS	435ms	3,730
stat_file	PASS	301ms	-

Compression A/B Tests

Metric	Value
Files tested	25
Total raw bytes	140,449
Total strip bytes	92,794
Total raw tokens	67,807
Total strip tokens	43,315
Tokens saved	24,492
Byte reduction	33%
Token reduction	36%
Min threshold	10%
Cache consistent	Yes

Per-file results

File	Raw B	Strip B	Raw Tok	Strip Tok	Tok Saved	Red%	Raw ms	Strip ms
2026-03-06_Collin_Stewart_Your_outbou...	10,293	10,218	2,337	2,316	21	0%	417ms	312ms
2026-03-06_sweetgreen_Feta_together_1...	6,088	6,053	4,181	4,171	10	0%	431ms	299ms
2026-03-06_Robert_from_Laminar_Welcom...	999	982	291	284	7	1%	439ms	287ms
2024-10-15_Harris-Walz_Organizing_Tea...	2,337	2,305	976	967	9	1%	398ms	299ms
2024-10-15_Stephen_King_via_KamalaHar...	2,904	2,866	1,116	1,103	13	1%	428ms	311ms
2026-03-06_Netflix_Harry_Styles._One_...	4,378	4,251	1,978	1,931	47	2%	425ms	317ms
2026-03-06_Sentry_Re_INFRA-A7_-_Cron_...	720	692	272	262	10	3%	420ms	299ms
2026-03-06_PagerDuty_PagerDuty_ALERT_...	1,476	1,429	485	466	19	3%	408ms	304ms
2025-07-17_KAYAK_How_to_have_everythi...	1,724	1,670	1,220	1,204	16	3%	461ms	293ms
2026-03-06_Eat_Clean_Bro_Reach_Your_G...	901	859	309	296	13	4%	417ms	290ms
2025-07-17_Alamo_Drafthouse_Cinema_Th...	19,047	18,224	12,228	11,704	524	4%	518ms	389ms
2025-07-17_Alamo_Drafthouse_Cinema_Th...	18,978	18,155	12,105	11,578	527	4%	517ms	406ms
2026-03-06_Sentry_BETA9-FRONTEND-DX_-...	865	785	325	299	26	9%	417ms	282ms
2026-03-06_New_York_State_DMV_eServic...	2,134	1,803	577	470	107	15%	416ms	289ms
2026-03-06_Sentry_INFRA-A7_-_Cron_fai...	1,115	852	381	292	89	23%	430ms	285ms
2026-03-06_DIRKS_LUCAS_M_Attention_AT...	3,730	2,821	1,071	620	451	24%	307ms	294ms
2026-03-06_The_OnBoard_Team_Watch_OnB...	2,073	1,496	731	387	344	27%	424ms	314ms
2026-03-06_Nat_Geo_Little_Kids_Privat...	2,918	2,056	1,095	501	594	29%	439ms	295ms
2026-03-06_Product_Hunt_Daily_ChatGPT...	12,900	7,958	3,502	2,066	1,436	38%	500ms	323ms
2024-10-15_Knut_at_Sanity_Join_us_tom...	1,272	726	504	209	295	42%	441ms	306ms
2025-07-17_Lindsay_Stanley_TODAY_Conf...	1,876	1,049	657	268	389	44%	404ms	286ms
2026-03-06_Thrive_Market_Just_for_you...	5,168	2,348	2,718	721	1,997	54%	445ms	295ms
2026-03-06_Kendall_Matthews_Re_Outbou...	3,809	704	981	210	771	81%	414ms	295ms
2026-03-06_Emily_from_Fever_Dino_Safa...	7,849	1,157	4,844	493	4,351	85%	421ms	288ms
2025-07-17_NORDSTROM_RACK_FLASH_Travi...	24,895	1,335	12,923	497	12,426	94%	555ms	326ms

Charts

Overall Compression	Raw vs Compressed Bytes	Token Comparison
Reduction by File	Read Latency	I/O Latency

[cubic-dev-ai]

1 issue found across 17 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="pkg/worker/usage_parser.go">

<violation number="1" location="pkg/worker/usage_parser.go:58">
P2: Snapshot discards partial JSON lines by clearing the buffer before confirming it can be parsed, so mid-stream snapshots can permanently lose usage data. Only clear the buffer when the trailing fragment is a valid JSON object (or keep it if unparseable).</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}