feat: add write coalescing with configurable batch flush#2
Open
jgwesterlund wants to merge 10 commits intomainfrom
Open
feat: add write coalescing with configurable batch flush#2jgwesterlund wants to merge 10 commits intomainfrom
jgwesterlund wants to merge 10 commits intomainfrom
Conversation
Merge pull request #1 from jgwesterlund/fix/ring-open-recovery-safety
lets hope, cause im tired....
Add coalesce_max_records and coalesce_max_seconds parameters to buffer writes and flush in batches. Initialize _last_flush_time to time.monotonic() (not 0.0) so time-based flush works from the start. Add flush(), write_stats, and include_pending parameter to iter_records(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
What does this change do?
Adds write coalescing to buffer multiple appends and flush them in batches. New constructor params
coalesce_max_recordsandcoalesce_max_secondscontrol flush thresholds. Addsflush(),write_statsproperty, andinclude_pendingparam oniter_records().Why is this needed?
Reduces I/O overhead for high-throughput workloads by batching writes instead of flushing per-record. Also fixes a bug where
_last_flush_timewas initialized to0.0, which meant a config with onlycoalesce_max_secondswould never trigger time-based flush (the elapsed check required_last_flush_time > 0). Now initialized totime.monotonic().Type of change
How was this tested?
7 new tests in
tests/test_extended.py:close()flushes pending recordsflush()explicit callwrite_statsaccuracyiter_records()default excludes pending;include_pending=Trueincludes themAll 43 tests pass, 99% coverage, ruff clean, mypy clean.
Checklist
Notes for reviewers
iter_records()defaults toinclude_pending=Falseso it only reflects durable on-disk state. Pending records are opt-in to avoid silently changing on-disk semantics (addressing reviewer concern from original PR).append()returns a projected sequence number for unflushed records since the actual sequence is assigned on flush.Docs updated:
usage.md,api.md.