Optimization: Bulk Enrichment of IOCs

[rootp1]

Hi @regulartim,
I had a thought on a safe optimization we can queue up, in which right now the IOC_from_hits path is functionally right, but it still queries per unique IP.
So even when we process a huge batch efficiently in memory, DB calls still grow as 3 + 4U + 1 (U = new src_ip). That’s feels expensive. We can keep all current behavior exactly the same and just shift enrichment internals to batch operations so we stop repeating the same lookups per IOC.

If we do that, query growth drops to about 6 + U (plus a small bounded M2M thing), which is a big improvement.

Like for Example at 1,000 unique ips, we go from roughly 4,004 queries to ~1,006, so about a 75% reduction with no intended change in output.
The practical win is better throughput headroom, more stable worker latency, and less DB pressure during spikes

We can do it by writing one optimized enrichment path.
Please share your thoughts on it

[regulartim]

Hey @rootp1 ! Thanks for bringing this up! :) I think we can do that if (and only if) it does not add to much complexity to the process. What do you mean exactly, when you talk about the "enrichment internals / path"? I am not quite sure what your approach for solving this would look like.

[rootp1]

Hey @regulartim, thanks for the clarification.

I see your point,

What do you mean exactly, when you talk about the "enrichment internals / path"?

By “enrichment internals/path”, I was referring to the lookup/create work that happens. The extraction logic itself would remain unchanged. same grouping by src_ip, same filtering, same resulting fields and output, which is all happening quite great already

I am not quite sure what your approach for solving this would look like.

existing idea is somethin' like:

1. first collect the unique ips or asns for the current batch up front
2. then fetch enrichment data at once
3. Building inmemory maps for those results
4. Attach the same enrichment values while creating each ioc, instead of rechecking the db per IOC

I guess existing implementation as above mentioned is already so strong

[rahulgunwanistudy-2005]

Hi @regulartim! I looked into this as well.

After tracing the full extraction pipeline, the bottlenecks are not quite where they appear:

Already efficient:

• FireHol exact match and MassScanner lookups in iocs_from_hits() are already bulk prefetched
• ASRepository preloads all AS records into memory at init, so there are no per-IOC DB calls there
• iocs_from_hits() itself only builds IOC objects in memory, no DB lookups per IP

The queries that actually add up:

1. IocProcessor.add_ioc() calls ioc_repo.get_ioc_by_name() once per IOC. That is one SELECT per unique IP, so U unique IPs equals U queries. This is the main bottleneck.

2. FireHolList.filter(ip_address__contains="/") in iocs_from_hits() loads ALL CIDR rows on every call. With multiple honeypots per chunk, this runs several times returning the same data each time.

3. _add_fks() in BaseExtractionStrategy calls get_ioc_by_name() twice for IOCs that add_ioc() just processed and already has in hand.

My Proposed fix:
Add _ioc_cache: dict[str, IOC] to IocProcessor with a prefetch_iocs(names) method that bulk-loads existing IOCs in one query before the per-IOC loop. After each save(), the cache gets updated. This turns U SELECTs into one bulk SELECT.

Add _extract_iocs_from_hits() to BaseExtractionStrategy that wraps iocs_from_hits() and calls prefetch_iocs() immediately after. Every strategy then changes one word: iocs_from_hits(hits) becomes self._extract_iocs_from_hits(hits).

This also fixes _add_fks() for free since both lookups become cache hits. It also removes Tanner's manual ioc_cache in _classify_attacks(), which is doing the same thing locally but only for that one strategy.

I have a PR (#1188) currently under review. Once that is merged I would be happy to implement this!

[regulartim]

Sounds good @rahulgunwanistudy-2005 ! I'll assign you.