Fix streaming dedup: keep last message.id occurrence for accurate tokens and tools#201
Merged
Fix streaming dedup: keep last message.id occurrence for accurate tokens and tools#201
Conversation
…ession files Claude Code writes the same message.id multiple times during streaming. The first write has partial tokens (often 1) and no tool_use blocks. The last write has authoritative token counts and all tool_use/MCP blocks. Old behavior kept the first occurrence (keep-first), silently dropping real output tokens (+6.3% undercount) and all MCP tool calls. New behavior keeps the last occurrence's content but preserves the first occurrence's timestamp for correct date bucketing. Validated against 21,390 real session files: 40.5% had duplicate IDs, output tokens were understated by up to 78% per session.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
message.idmultiple times during streaming (message_start, intermediate, message_stop)Validation (4 agents, 21,390 real session files)
Before/After comparison (6 days, Claude only)
Old codeburn matched ccusage within 0.05% because both had the same keep-first bug.
Scope
Only affects Claude Code session parsing. Other providers (Gemini, Cursor, Goose, etc.) have their own parsers and are unaffected.
Closes #110