(retriever) Add .store() task for persisting extracted images (#1675)#1714
Open
(retriever) Add .store() task for persisting extracted images (#1675)#1714
Conversation
- Add store_extracted_images() with fsspec/UPath support for local and cloud storage - Wire .store() into InProcessIngestor, BatchIngestor, and batch_pipeline CLI - Add StoreParams model, unit tests, and fsspec/universal-pathlib deps Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
…mn name fix - Fix column names (table/chart/infographic not plural) to match OCR output - Add magic byte sniffing so file extension matches actual image encoding - Make base64 stripping opt-in (strip_base64=False) to preserve multimodal compat - Add multimodal embed interaction tests and format consistency tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
storage_uri to absolute Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
Signed-off-by: Jacob Ioffe <jioffe@nvidia.com>
jperez999
reviewed
Mar 30, 2026
Collaborator
jperez999
left a comment
jperez999
left a comment
There was a problem hiding this comment.
Would love to see an actual graph unit test for this store behavior. Perhaps we could run a few of the stages in a graph locally get to the OCR stage and export. Might be worth having tests that show we can export at multiple stages in the pipeline and get the things we are targeting at each store call.
| lancedb: LanceDbParams = Field(default_factory=LanceDbParams) | ||
|
|
||
|
|
||
| class StoreParams(_ParamsModel): |
Collaborator
There was a problem hiding this comment.
Does this allow me to store full chunks also... I think that is the only permutation you dont have here, text..?
| from nemo_retriever.params import StoreParams | ||
|
|
||
|
|
||
| def _make_tiny_png_b64(width: int = 4, height: int = 4, color=(255, 0, 0)) -> str: |
Collaborator
There was a problem hiding this comment.
Some of these functions are doubled up in other files. We are going to need to consolidate that.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
.store()pipeline task tonemo_retrieverthat persists extracted images (full-page + sub-page table/chart/infographic/image assets) to local or cloud storage via fsspec/UPath..store()intoInProcessIngestor,BatchIngestor(map_batches), and batch CLI via--store-images-uriStoreParamswith per-content toggles,image_format, andstrip_base64(defaultFalse— opt-in memory reduction, preserves multimodal embedding compatibility)encodingfield + magic-byte sniff fallback); cropped outputs follow configuredimage_formatpublic_base_url,storage_options) available viaStoreParamsin library use; CLI exposes--store-images-urifor common caseTest plan
test_io_image_store.py: page images, structured crops, natural images, format consistency, stripping, edge casestest_multimodal_embed.py: verifies store → embed behavior for multimodal with/without strippingbo20dataset (20 PDFs, 496 pages) → 842 stored assets (496 page images + 346 table/chart crops)Known limitations (follow-ups)
table,chart,infographic) matching OCR stage outputimage_b64skip: if directimage_b64exists but decode fails, we skip rather than falling back tobbox_xyxy_normcrop from page image_safe_stemcollision: two files with same basename but different directories share output subdirectory; hash-based naming is a follow-upCloses #1675
🤖 Generated with Claude Code