feat: document processing #4
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Replace invalid temp_dict.add() usage with a dict literal to fix runtime errors. - Pass section_stack into traverse_blocks to ensure proper block hierarchy traversal. - Pass db parameter into insert_content to allow database access where needed. - Remove dead or commented-out config_parser code to clean up unused logic.
…nto a context class
… speedup - Added content_buffer to DocumentProcessingContext - Modified insert_content to buffer KnowledgeBaseContent objects - Used bulk_save_objects for batched database insertions - Improved processing speed for large PDFs and textbooks
- Implemented dynamic buffering for database inserts - Auto-flushes content when buffer fills to maintain constant memory usage - Enables smooth streaming ingestion for multi-thousand page documents without stalls
feat: document processing This PR significantly improves the performance and scalability of document ingestion in the document_processing module. # Key changes: Introduced content_buffer to DocumentProcessingContext for temporary batching of KnowledgeBaseContent objects. Reworked insert_content to use a dynamic buffer that auto-flushes to the database when full. Implemented bulk_save_objects for fast batched database insertions, replacing individual add/commit calls. Ensured constant memory usage during ingestion to handle multi-thousand page documents without stalls or blowups.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Key changes:
Introduced content_buffer to DocumentProcessingContext for temporary batching of KnowledgeBaseContent objects.
Reworked insert_content to use a dynamic buffer that auto-flushes to the database when full.
Implemented bulk_save_objects for fast batched database insertions, replacing individual add/commit calls.
Ensured constant memory usage during ingestion to handle multi-thousand page documents without stalls or blowups.