fix(astro): mirror batch-size override and ECONNRESET retry to full-reindex script#134
Merged
chris-c-thomas merged 1 commit intomainfrom Apr 25, 2026
Merged
Conversation
…eindex script apps/astro/scripts/index-search.ts gets the same treatment that index-search-incremental.ts received in #133: - New --batch-size <n> CLI flag and MEILI_BATCH_SIZE env var override the default of 500 docs/batch. - New --verbose-batches flag prints first/last doc ID per flushed batch with stdout force-flushed so the last logged ID survives a crash. - New flushWithRetry() in BatchIndexer waits for /health to return "available" (up to 180s) and retries on the original taskUid rather than resubmitting. Up to 5 attempts per flush. apps/astro/CLAUDE.md says these two scripts must be kept in sync — this PR closes the drift introduced when #133 only updated the incremental sibling. Behavior identical to #133. Tested: - npx tsx scripts/index-search.ts --batch-size abc → rejects non-integer - MEILI_BATCH_SIZE=xyz npx tsx scripts/index-search.ts → rejects non-integer - pnpm turbo build:astro → passes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Mirror of #133 to the full-reindex sibling.
apps/astro/scripts/index-search.tsnow has the same surface asindex-search-incremental.ts:--batch-size <n>CLI flag andMEILI_BATCH_SIZEenv var override the default 500 docs/batch--verbose-batcheslogs first/last doc ID of every flushed batch (stdout force-flushed)flushWithRetry()handles ECONNRESET during Meilisearch OOM crashes by waiting on/healthand retrying on the originaltaskUidWhy
apps/astro/CLAUDE.mdsays these two scripts must be kept in sync — same sources, sameSearchDocumentshape, sameconfigureIndexsettings. #133 only updated the incremental sibling because that's the script that was field-validated on the VPS. This PR closes the drift before it bites someone running a full reindex.The full reindex is rarely run, but when it is run it's the operation most exposed to OOM (it writes ~1M docs in one pass with no per-source checkpointing). The retry path is exactly the kind of thing you want already in place by the time you need it.
Test plan
npx tsx scripts/index-search.ts --batch-size abcrejects non-integerMEILI_BATCH_SIZE=xyz npx tsx scripts/index-search.tsrejects non-integerpnpm turbo build:astropassesgray-matter cache) unrelated and unchanged--batch-size 100 --verbose-batcheson the next full rebuild to confirm the retry path