Enhancement: bulk prefetch IOCs before processing to eliminate per-IOC SELECT queries. Closes #1241#1296
Closed
rahulgunwanistudy-2005 wants to merge 17 commits intoGreedyBear-Project:mainfrom
Conversation
…oses GreedyBear-Project#1234 (GreedyBear-Project#1258) Signed-off-by: Drona Raj Gyawali <dronarajgyawali@gmail.com>
…ct#1280 (GreedyBear-Project#1286) * Add rule set * Fix import
…oject#1280 (GreedyBear-Project#1287) * Add ruel set * Fix violations
…reedyBear-Project#1280 (GreedyBear-Project#1288) * Add rule set * Add rule set * Add rule set * Add ruff rule FA
…Bear-Project#1284) * Install gb-ui library * Replace imports * Update frontend README
…ject#1280 (GreedyBear-Project#1290) * Add rule set * Fix violations
…ect#1280 (GreedyBear-Project#1291) * Add ruff rule * Fix violations * Exclude SIM108 rule
…t#1280 (GreedyBear-Project#1292) * Add rule and exception for G004 * Reorder ignores * Fix violations
…ect#1280 (GreedyBear-Project#1293) * Add rule * Fix violations
…1260 - Reduced Tim… (GreedyBear-Project#1289) * Enhancement: optimization in Cowrie _get_sessions GreedyBear-Project#1260 - Reduced Time Complexity from O(N*M) to O(M) * chore: simplify src_ip extraction in _get_scanners to fix PR review * chore: remove redundant src_ip check per reviewer feedback
…r-Project#1223 (GreedyBear-Project#1250) * Fix statistics source handling for proxy and IPv6 * Add merge migration for conflicting greedybear 0050 leaves * Make statistics source migration sequential after 0050 * Set 0051 statistics migration dependency to 0050_attackeractivitybucket * Address moderator feedback: raise exception instead of returning empty string - Add UnableToExtractSourceIPError custom exception - Modify get_request_source_ip() to raise exception with logging when no valid IP found - Update all callers (utils.py, enrichment.py, command_sequence.py, cowrie_session.py) to handle exception - Update test to expect exception behavior instead of empty string - Statistics recording is now skipped when source IP cannot be extracted * Fix formatting: add blank line after docstring * Fix linter errors: remove unnecessary pass and sort imports * Fix import order in command_sequence.py to match original
…ct#1278 (GreedyBear-Project#1279) * add database index for attacker_country_code * generate migration for attacker_country_code index * normalize attacker_country_code to uppercase on write * use exact lookup with upper() for country_code query * fix migration chain after merging develop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The extraction pipeline was calling
ioc_repo.get_ioc_by_name()once per IOC insideadd_ioc(), so a batch of U unique IPs meant U SELECT queries. On top of that,_add_fks()inBaseExtractionStrategywas re-fetching IOCs thatadd_ioc()had just processed. Tanner had a manual localioc_cachein_classify_attacks()as a partial workaround for the same problem.This PR fixes all three by introducing a bulk prefetch mechanism in
IocProcessor:_ioc_cacheandprefetch_iocs()toIocProcessor. Each_get_scanners()now bulk-loads all IOCs for the batch in one query before the loop, reducing U SELECTs to 1.add_ioc()checks the cache first, falls back to the repo only on a miss, and updates the cache after every save._add_fks()now builds an in-memory map fromself.ioc_recordsinstead of making extra DB calls for IOCs already processed.ioc_cachein_classify_attacks()is removed and replaced by the processor-level cache.Related issues
Closes #1241
Type of change
Checklist
Formalities
Enhancement: bulk prefetch IOCs before processing to eliminate per-IOC SELECT queries. Closes #1241develop.develop.Docs and tests
GUI changes
No GUI changes in this PR.