Debug: Anthropic cache token cost investigation#44
Conversation
Reproduction script investigating why $ai_input_tokens can be inclusive (input + cache_read) instead of exclusive, causing cost overcharges. Tests raw API, SDK wrapper, and posthog_properties override behavior. Writes results to output.md for sharing.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 711f5cda6e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| prev_api_input = response.usage.input_tokens | ||
| prev_cache_read = getattr(response.usage, "cache_read_input_tokens", 0) or 0 | ||
| prev_output = response.usage.output_tokens | ||
| inclusive_input = prev_api_input + prev_cache_read | ||
| computed_total = inclusive_input + prev_output |
There was a problem hiding this comment.
Step 3b derives inclusive_input and computed_total from response (the previous 3a call) and then applies them to a new messages.create() request with a different prompt; if token usage changes between calls, the reported “correct vs overcharged” comparison is no longer apples-to-apples and the overcharge factor becomes inaccurate. This can mislead the investigation output, so both exclusive and overridden values should be tied to the same request context.
Useful? React with 👍 / 👎.
1 participant
Summary
input_tokens, confirms the PostHog SDK passes them through correctly, and demonstrates howposthog_propertiesoverrides can cause the bugoutput.mdfor easy sharingContext
Customer reported ~7x cost overcharge on Anthropic cached calls. Investigation found the SDK is correct but
posthog_propertiescan override the SDK's exclusive token values with inclusive ones.See
python/scripts/debug-anthropic-cache-cost/output.mdfor full results.Test plan
output.mdgenerated with formatted results