Rippleitinnz-bloom-filtering-patch1 [DO NOT MERGE]#405
Open
rippleitinnz wants to merge 16 commits intoEvernodeXRPL:mainfrom
Open
Rippleitinnz-bloom-filtering-patch1 [DO NOT MERGE]#405rippleitinnz wants to merge 16 commits intoEvernodeXRPL:mainfrom
rippleitinnz wants to merge 16 commits intoEvernodeXRPL:mainfrom
Conversation
Increase MAX_IN_MSG_QUEUE
Increase MAX_QUEUE_SIZE size to handle larger UNL
increase MAX_NPL_MSG_QUEUE_SIZE and MAX_CONTROL_MSG_QUEUE_SIZE
Updating deduplication to bloom filtering
moving deduplication to bloom filters
Moving deduplication to bloom filters
moving deduplication to bloom filters
moving deduplication to bloom filters
Moving deduplication to bloom filters
moving deduplication to bloom filters
moving deduplication to bloom filters
moving deduplication to bloom filters
RichardAH
reviewed
Jun 21, 2025
RichardAH
reviewed
Jun 21, 2025
| util::rollover_hashset recent_peermsg_hashes(200); | ||
|
|
||
| /** | ||
| /** |
Collaborator
There was a problem hiding this comment.
where's the actual use of the bloom filter to deduplicate? you probably need to run two at the same time so that periodically you can clear one or the other to prevent them filling up and becoming useless
Contributor
Author
There was a problem hiding this comment.
The bloom filter implementation maintains the exact same interface (try_emplace) so no other code changes are needed. The global recent_peermsg_hashes is now defined in bloom_filter.hpp and works exactly the same way.
Will look at running two bloom filters at the same time and will amend
Added in rolling bloom filter. Two 16MB filters (32MB total) that rotate every 5 minutes When checking, we look in both filters. When inserting, we add to both. Every 5 minutes, the older filter is cleared and becomes the new filter Since we insert into both filters, messages are retained for 5-10 minutes Uses atomic operations to ensure only one thread performs rotation The behaviour is: Minutes 0-5: Filter 1 is active, Filter 2 is building Minute 5: Clear Filter 1, switch to Filter 2 as active Minutes 5-10: Filter 2 is active, Filter 1 is building Minute 10: Clear Filter 2, switch to Filter 1 as active and so on...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moving deduplication to bloom filtering.
Key changes:
Created a bloom_filter class that uses 32MB of memory
Added a typedef using rollover_hashset = bloom_filter; for compatibility
Created global instances for recent_peermsg_hashes and recent_selfmsg_hashes
All files now include bloom_filter.hpp instead of rollover_hashset.hpp
Removed local declarations of the hash sets since they're now global
This maintains full compatibility with the existing code while switching to the bloom filter implementation.