Skip to content

feat(logbuf): Cisco-style "last message repeated N times" dedup#26

Merged
arcaven merged 1 commit intomainfrom
feat/logbuf-dedup
Apr 19, 2026
Merged

feat(logbuf): Cisco-style "last message repeated N times" dedup#26
arcaven merged 1 commit intomainfrom
feat/logbuf-dedup

Conversation

@arcaven
Copy link
Copy Markdown
Collaborator

@arcaven arcaven commented Apr 19, 2026

Summary

Closes the symptom Skippy reported in aae-orc-1d2: a single "ssh: client connected" line at 0.5 Hz dominated the daemon's 10k-line ring buffer, shrinking the visible-history window from days to ~5.5 hours. Cisco-style "last message repeated N times" dedup collapses adjacent identical lines into one stored slot plus a counter — a 1000-line run consumes 2 ring slots, not 1000.

Behavior

Before:                       After:
ssh: client connected         ssh: client connected
ssh: client connected         last message repeated 999 times so far
ssh: client connected         (next interesting log line)
... (998 more) ...
(next interesting log line)

When a different line arrives, the buffer emits a last message repeated N times summary before recording the new line. Tail() synthesizes a ... so far summary for any still-active run so an operator querying mid-burst sees it without waiting. Flush() is the periodic-cut hook for runs that never see a different line; not wired to a timer in this PR (caller decides).

Design choices

Captured in _kos/ideas/log-rrd-deduplication.md (the aae-orc-4wz brainstorm):

  • Previous-line-only matching (Cisco original). A B A B does not collapse — documented in TestDedup_NonAdjacentNotDeduped.
  • No regex-based key normalization in this pass. Lines must match exactly. The three motivating cases (ssh poll, crashloop chatter, future debug spam) are all monotonic, so exact-match catches them all without the false-collapse risk of regex normalization.
  • Synthesized active-run summary in Tail() rather than a stored "live" summary that would have to be replaced on each new repeat. Keeps the stored buffer clean and Flush() simple.
  • Flush is idempotent so a periodic timer wiring is safe with no was-active bookkeeping at the call site.

Test plan

  • Existing tests still pass (TestWrite_LargeSinglePayload updated — 100 identical writes now stored as 1 line + active count, was 3 lines bound to cap)
  • TestDedup_RunBrokenBySummary — A A A B → [A, "repeated 2 times", B]
  • TestDedup_NonAdjacentNotDeduped — A B A B → no collapse
  • TestDedup_TailSynthesizesActiveSummary — A A A A → [A, "repeated 3 times so far"]
  • TestDedup_FlushEmitsSummary — Flush appends summary and clears active run
  • TestDedup_FlushIdempotent — second Flush is no-op
  • TestDedup_SingleOccurrenceNoSummary — A B → no spurious "repeated 0 times"
  • TestConcurrentWrites still passes with -race
  • golangci-lint clean

Related deferred work

aae-orc-4wz (the broader RRD-style probe with key extraction, structured repeat_count in the logs RPC result, OTEL bridge) stays open. This PR ships the smallest version that makes the concrete 1d2 symptom go away.

Refs: aae-orc-1d2, aae-orc-4wz

Adjacent identical lines are collapsed into a single stored slot plus
an in-memory counter. When a different line arrives (or Flush is
called) the buffer emits a "last message repeated N times" summary
before recording the new line. Tail() synthesizes a "... so far"
summary for any active run so an operator querying mid-burst sees it
without waiting for a flush.

Closes the symptom Skippy reported in aae-orc-1d2: the desk monitor
loop spammed one "ssh: client connected" line at 0.5 Hz, dominating
the 10k-line ring and shrinking the visible-history window from
days to ~5.5 hours. Dedup turns a 1000-line run into 2 ring slots
without hiding the rate.

Design choices and tradeoffs captured in
marvel/_kos/ideas/log-rrd-deduplication.md (the 4wz brainstorm).
- Previous-line-only matching (Cisco original); A B A B does not
  collapse. Documented in test.
- No regex-based key normalization in this pass; lines must match
  exactly. The session-026 evidence is monotonic so this catches
  all three observed motivators.
- Flush() is the periodic-cut hook for very long runs that never
  see a different line. Caller wires it into a timer when desired.

Refs: aae-orc-1d2, aae-orc-4wz
@arcaven arcaven merged commit 7f9c387 into main Apr 19, 2026
7 checks passed
@arcaven arcaven added type.feature Net-new capability agent.worker PR created by a Claude Code worker agent area.logs logbuf labels Apr 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent.worker PR created by a Claude Code worker agent area.logs logbuf type.feature Net-new capability

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant