feat: restore git mirrors from S3 snapshots on cold start#170
Merged
Conversation
Allows clients to request shallow snapshots at a specific git depth
(e.g., /git/{repo}/snapshot.tar.zst?depth=100). This produces much
smaller snapshots for large repositories — a depth-100 snapshot of a
multi-GB repo is typically under 1GB compressed, versus 13GB+ for a
full snapshot.
The depth parameter controls the git clone depth used when generating
the snapshot. Full snapshots (no depth parameter) continue to work as
before. Each requested depth gets its own cache key and periodic refresh
job, so snapshots stay fresh without rebuilding depths that aren't used.
On first request for a given depth, the snapshot is generated on-demand
and a periodic refresh job is scheduled. Subsequent requests serve from
cache.
worstell
force-pushed
the
worstell/snapshot-restore-cold-start
branch
from
ad8db59 to
e224296
Compare
inez
reviewed
Mar 10, 2026
Contributor
inez
left a comment
inez
left a comment
There was a problem hiding this comment.
Workstations seems to currently try to clone with depth=100 but here, if I understand correctly, restored snapshot (if exist) would be of depth=0?
Contributor
Author
depth 0 is full depth, which is what workstations request from cachew too. before caching we were using depth-limited cloning but we get full clones now that said the depth stuff was a vestige of a prior approach we didnt decide to move forward with (supporting depth-limited snapshots) so i just removed it entirely |
worstell
force-pushed
the
worstell/snapshot-restore-cold-start
branch
from
e224296 to
71364e3
Compare
alecthomas
approved these changes
Mar 10, 2026
| // tryRestoreSnapshot attempts to restore a mirror from an S3 snapshot. Returns | ||
| // true if the restore succeeded and the repo is ready to serve. | ||
| func (s *Strategy) tryRestoreSnapshot(ctx context.Context, repo *gitclone.Repository) bool { | ||
| if s.cache == nil { |
Contributor
Author
There was a problem hiding this comment.
no this was a guard amp added to get tests to succeed. its updating with a test cache instead
worstell
force-pushed
the
worstell/snapshot-restore-cold-start
branch
from
71364e3 to
a068c87
Compare
Cold-starting pods (new/restarted/scaled) previously had to run a full git clone --mirror while proxying all requests to GitHub, which takes minutes for large repos. S3 snapshots already exist (created periodically by warm pods) but were never used during cold start. startClone() now attempts snapshot.Restore() from the tiered cache before falling back to git clone --mirror. On success, a catch-up fetch is scheduled via the job scheduler to cover any staleness. Changes: - Add Repository.MarkRestored() to transition StateEmpty -> StateReady after an external restore, applying configureMirror and registerMaintenance (matching Clone's behavior). - Add Strategy.tryRestoreSnapshot() which downloads and extracts the depth-0 snapshot, then calls MarkRestored. On any failure, cleans up and returns false so startClone falls through to the existing clone path. Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019cd9c4-9869-75b1-bc83-50484949b25b
worstell
force-pushed
the
worstell/snapshot-restore-cold-start
branch
from
a068c87 to
f9b5bf4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Cold-starting pods (new/restarted/scaled) have no local git mirror. They proxy all requests to GitHub while running a background
git clone --mirror, which takes minutes for large repos. S3 snapshots already exist (created periodically by warm pods) but are never used during cold start.Solution
startClone()now attemptssnapshot.Restore()from the tiered cache before falling back togit clone --mirror. On success, a catch-up fetch is scheduled via the job scheduler to cover any staleness from the snapshot interval.Changes
internal/gitclone/manager.goRepository.MarkRestored(ctx)— transitionsStateEmpty → StateReadyafter an external restore. AppliesconfigureMirror(bitmap/MIDX/commit-graph/pack tuning) andregisterMaintenance, matchingClone()'s behavior and locking protocol. Reverts toStateEmptyon failure.internal/strategy/git/git.goStrategy.tryRestoreSnapshot()— downloads and extracts the depth-0 snapshot viasnapshot.Restore, then callsMarkRestored. On any failure, cleans up the path and returns false sostartClonefalls through to the existing clone path.startClone()— tries snapshot restore first. On success: cleans up spools, schedules a catch-up fetch, and schedules periodic snapshot/repack jobs. On failure: falls through to the existinggit clone --mirrorpath unchanged.internal/strategy/git/snapshot.gonolint:gosecannotations for pre-existingos.RemoveAll/os.MkdirAllcalls on controlled paths (surfaced by the newsnapshotimport).What stays the same
serveWithSpoolstill handles in-flight requests during restore