Reduce Memory footprint for various functionalities

[ss4328]

Currently, all operations are performed in-memory. Let's batch process it to make the implementation more memory friendly.

Some ideas:

• Files breaching a certain threshold will trigger chunking
• TQDM for keeping user in the loop
• Validation of available space for uber-large files. Let's say 4GB to start with, tunable via config.py

[onestardao]

yo, love the direction here — especially chunking based on file size threshold.
we actually ran into similar bottlenecks when trying to do in-memory parsing at scale, and ended up building a semantic-aware chunking + streaming layer that auto-manages memory thresholds and preserves context during batch splits.

mapped a bunch of these scenarios (memory, chunk drift, async race) into a failure guide, MIT-licensed, tested with real-world deployments.
if it’s useful to you i can send it over, just ping me.

might save you hours of fiddling with config.py and TQDM loops 😅

[ss4328]

Hey @onestardao, I appreciate the interest and the information volunteering! Please send the info over, and I'll have a look!

[onestardao]

yo, love the batching + threshold strategy — it’s heading in exactly the right direction.

just a heads-up: a lot of RAG systems hit subtle failures when doing large file ingestion in-memory. from what you're describing, you're likely brushing against at least two of the core failure patterns we've documented here:

Problem Map – 16 failure patterns in real-world AI apps

the ones that might be most relevant to your use case:

No.7: Memory Breaks Across Sessions – memory gets dropped or misaligned across stream chunks

No.14: Bootstrap Ordering – things fire before deps (e.g., retriever or schema) are ready

No.15: Deployment Deadlock – circular waits due to index/DB order mismatches

i’ve run into these before and ended up building a semantic-aware memory layer that adapts to batch size, guards index readiness, and defers LLM load until schema + ingestion is confirmed.

if you’re curious or want a drop-in tool to test against these failure modes, just let me know. open source + MIT license. might save you a few weekends