fix: save marker after Chronicle delivery to prevent data loss on crash#25
Merged
carlosmmatos merged 1 commit intomainfrom Apr 17, 2026
Merged
fix: save marker after Chronicle delivery to prevent data loss on crash#25carlosmmatos merged 1 commit intomainfrom
carlosmmatos merged 1 commit intomainfrom
Conversation
State persistence was in the reader thread, saving the marker after each fetch cycle regardless of whether indicators were delivered to Chronicle. If the process crashed between fetching and sending, the marker would advance past unsent indicators, creating a gap on restart. Moves state persistence to the writer thread so the marker is only saved after a batch is successfully delivered to Chronicle. The queue now carries (indicators, marker) tuples so the writer knows which marker to persist. Also stops the writer from continuing to send remaining batches when a batch permanently fails after 30 retries. Additionally fixes the cold-start FQL filter — the reader was using `_marker:>=` with a float timestamp, which doesn't filter against opaque marker strings, causing the API to return the full corpus (~5.5M indicators) regardless of `initial_sync_lookback`. Now uses `last_updated:>=` for numeric timestamps and `_marker:>=` for real marker strings. Excludes config/devel.ini from Docker builds via .dockerignore to prevent local dev overrides from leaking into container images.
redhatrises
approved these changes
Apr 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
State persistence was in the reader thread, saving the
_markercursor after each fetch cycle regardless of whether indicators had actually been delivered to Chronicle. If the process crashed between fetching and writing, the marker would advance past unsent indicators, creating a silent data gap on restart.Moves state persistence to the writer thread so the marker is only saved after a batch is successfully delivered. The queue now carries
(indicators, marker)tuples so the writer knows which cursor to persist. The writer also stops sending remaining batches when one permanently fails after 30 retries, preventing partial-delivery marker advancement.Also fixes the cold-start FQL filter — the reader was building
_marker:>='1745007000.123'+deleted:falsewith a float timestamp on cold start. Since_markeris an opaque cursor string, this didn't filter meaningfully and the API returned the entire indicator corpus (~5.5M), makinginitial_sync_lookbackineffective. Now useslast_updated:>=for numeric timestamps and_marker:>=for real marker strings, with seamless transition during pagination.