feat(logbuf): Cisco-style "last message repeated N times" dedup#26
Merged
feat(logbuf): Cisco-style "last message repeated N times" dedup#26
Conversation
Adjacent identical lines are collapsed into a single stored slot plus an in-memory counter. When a different line arrives (or Flush is called) the buffer emits a "last message repeated N times" summary before recording the new line. Tail() synthesizes a "... so far" summary for any active run so an operator querying mid-burst sees it without waiting for a flush. Closes the symptom Skippy reported in aae-orc-1d2: the desk monitor loop spammed one "ssh: client connected" line at 0.5 Hz, dominating the 10k-line ring and shrinking the visible-history window from days to ~5.5 hours. Dedup turns a 1000-line run into 2 ring slots without hiding the rate. Design choices and tradeoffs captured in marvel/_kos/ideas/log-rrd-deduplication.md (the 4wz brainstorm). - Previous-line-only matching (Cisco original); A B A B does not collapse. Documented in test. - No regex-based key normalization in this pass; lines must match exactly. The session-026 evidence is monotonic so this catches all three observed motivators. - Flush() is the periodic-cut hook for very long runs that never see a different line. Caller wires it into a timer when desired. Refs: aae-orc-1d2, aae-orc-4wz
This was referenced Apr 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the symptom Skippy reported in aae-orc-1d2: a single "ssh: client connected" line at 0.5 Hz dominated the daemon's 10k-line ring buffer, shrinking the visible-history window from days to ~5.5 hours. Cisco-style "last message repeated N times" dedup collapses adjacent identical lines into one stored slot plus a counter — a 1000-line run consumes 2 ring slots, not 1000.
Behavior
When a different line arrives, the buffer emits a
last message repeated N timessummary before recording the new line.Tail()synthesizes a... so farsummary for any still-active run so an operator querying mid-burst sees it without waiting.Flush()is the periodic-cut hook for runs that never see a different line; not wired to a timer in this PR (caller decides).Design choices
Captured in
_kos/ideas/log-rrd-deduplication.md(the aae-orc-4wz brainstorm):A B A Bdoes not collapse — documented inTestDedup_NonAdjacentNotDeduped.Tail()rather than a stored "live" summary that would have to be replaced on each new repeat. Keeps the stored buffer clean and Flush() simple.was-activebookkeeping at the call site.Test plan
TestWrite_LargeSinglePayloadupdated — 100 identical writes now stored as 1 line + active count, was 3 lines bound to cap)TestDedup_RunBrokenBySummary— A A A B → [A, "repeated 2 times", B]TestDedup_NonAdjacentNotDeduped— A B A B → no collapseTestDedup_TailSynthesizesActiveSummary— A A A A → [A, "repeated 3 times so far"]TestDedup_FlushEmitsSummary— Flush appends summary and clears active runTestDedup_FlushIdempotent— second Flush is no-opTestDedup_SingleOccurrenceNoSummary— A B → no spurious "repeated 0 times"TestConcurrentWritesstill passes with-racegolangci-lintcleanRelated deferred work
aae-orc-4wz(the broader RRD-style probe with key extraction, structuredrepeat_countin the logs RPC result, OTEL bridge) stays open. This PR ships the smallest version that makes the concrete 1d2 symptom go away.Refs: aae-orc-1d2, aae-orc-4wz