fix(migrations): bump v0.12.0 backfill timeout from 10min to 30min#201
Open
orendi84 wants to merge 1 commit intogarrytan:masterfrom
Open
fix(migrations): bump v0.12.0 backfill timeout from 10min to 30min#201orendi84 wants to merge 1 commit intogarrytan:masterfrom
orendi84 wants to merge 1 commit intogarrytan:masterfrom
Conversation
Phase C (backfill_links) and D (backfill_timeline) both shelled out with a 10-minute timeout. On a 13k-page brain, each extraction takes ~11 minutes, so both phases returned 'failed' via execSync timeout and the orchestrator marked the migration 'partial'. Schema + verify phases were unaffected. Bump both timeouts to 30 minutes (1_800_000 ms). At the observed rate of ~50 sec/1k pages this covers brains up to ~35k pages; larger brains will need a proportional extension. Verified by re-running on a 13002-page brain: both phases complete cleanly in ~11 min each, migration marks 'complete'. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
phaseCBackfillLinksandphaseDBackfillTimelineinsrc/commands/migrations/v0_12_0.tseach wrapgbrain extract {links,timeline} --source dbin anexecSynccall withtimeout: 600_000(10 minutes).On a brain of non-trivial size that is too tight. On my 13,002-page brain, each extraction takes ~11 minutes - right past the cutoff. Both phases return
failed, the orchestrator marks the migrationpartial, and the daily post-upgrade hook keeps retrying the same thing and failing the same way every day.Fix
Bump both timeouts from
600_000(10 min) to1_800_000(30 min).At the observed rate of ~50 sec/1k pages this covers brains up to ~35k pages; larger brains will need a proportional extension, but 30 min is a far safer default than 10 min for any non-trivial corpus.
Verification
On a 13,002-page brain (127,093 chunks, all embedded):
Before the fix: migration consistently finished as
partial, both backfill phases timed out at ~10 min with no output written.After the fix: migration completes cleanly in ~22 min total. Both backfill phases run to completion (~11 min each), produce their summary lines, and verify phase runs after. Migration status:
complete.Scope
One-line change to two lines in one file. No new imports, no test changes needed (the existing
test/migrate.test.tsdoesn't exercise realexecSyncagainst prod-sized data so the timeout path isn't covered by tests today - consider that a follow-up).