fix: resolve git fetch/pull returning stale refs#192
Merged
Conversation
inez
reviewed
Mar 16, 2026
Comment on lines
-698
to
-699
| r.mu.RLock() | ||
| defer r.mu.RUnlock() |
Contributor
Author
There was a problem hiding this comment.
git handles its own file locking during repack so this wasn't needed
inez
reviewed
Mar 16, 2026
| cmd := exec.CommandContext(repackCtx, "git", "-C", r.path, "repack", "-d", "--geometric=2", "--write-midx", "--write-bitmap-index") | ||
| cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} | ||
| cmd.Cancel = func() error { | ||
| return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) |
Contributor
There was a problem hiding this comment.
Seems like this will be called always at the end (defer), but process might be already gone by then.
Contributor
Author
There was a problem hiding this comment.
Cancel only gets invoked while the process is still running so this shouldn't be an issue
Two fixes for git fetch/pull not working from workstations: 1. ensureRefsUpToDate now fetches directly when ls-remote confirms stale refs, bypassing the NeedsFetch time guard that was silently skipping the fetch even after staleness was detected. 2. Switch from full repack (-adb) to geometric repacking (--geometric=2). Full repack recompresses ALL objects and ran 15+ minutes on large repositories, blocking git http-backend. Geometric repacking only merges when small packs accumulate, completing near-instantly in steady state. A 10-minute timeout with process group kill is added as a safety net. Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019cf844-5ed1-74ae-bc57-27d068380f2a
worstell
force-pushed
the
fix-git-fetch-stale-refs
branch
from
1e6e75a to
eb8cf2a
Compare
submitFetch is called when ls-remote has already confirmed the mirror is stale. It was calling backgroundFetch which re-checks NeedsFetch(15m) and silently drops the fetch if one happened recently (e.g. post-restore). This caused the mirror to never catch up, making every request hit the upstream fallback path. Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019cf8a0-d586-7109-9e69-9cb7d9b27f6c
inez
approved these changes
Mar 16, 2026
| if err := s.backgroundFetch(ctx, repo); err != nil { | ||
| logger.WarnContext(ctx, "Synchronous fetch failed", "upstream", repo.UpstreamURL(), "error", err) | ||
| } | ||
| logging.FromContext(ctx).WarnContext(ctx, "Failed to check upstream refs", "error", err) |
Contributor
There was a problem hiding this comment.
Are you intentionally swallowing the error here?
Contributor
Author
There was a problem hiding this comment.
this should be returned, updated
jrobotham-square
approved these changes
Mar 16, 2026
Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-019cf8a0-d586-7109-9e69-9cb7d9b27f6c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Workstations running
git fetch/git pullthrough cachew either return stale data or hang.Root cause 1 — stale refs not forwarded: When
ls-remotedetects the mirror is behind upstream, the request was still served from the local mirror with stale refs. The background fetch meant to freshen the mirror calledbackgroundFetch, which re-checksNeedsFetch(15m)and silently skips if a fetch completed recently (e.g. post-restore on pod startup). The mirror never catches up.Root cause 2 — repack blocking:
git repack -adbrecompresses ALL objects into a single pack. On large repositories this runs 15+ minutes, during whichgit http-backend(upload-pack) hangs because pack files are being rewritten underneath it.Fix
Forward info/refs to upstream when refs are stale. When
ls-remoteconfirms the mirror is behind, the info/refs request is forwarded directly to upstream GitHub so the client gets fresh data immediately. A background fetch is scheduled to freshen the mirror for subsequent requests. The fetch bypasses theNeedsFetch(15m)time guard sincels-remotealready confirmed staleness. If the subsequent upload-pack hits the mirror before the fetch completes, the existing "not our ref" fallback forwards to upstream.Switch to geometric repacking (
--geometric=2 -d) instead of full repack (-adb). Geometric repacking only merges packs when small packs accumulate — near-instant in steady state vs 15+ minutes for full repack. A 10-minute timeout with process group kill is retained as a safety net.Keep synchronous fetch for snapshot requests. Workstation creation needs fresh data to generate correct snapshots, so the fetch remains synchronous on that path.