Skip to content

fix: cloning large repo's failed with fatal: early EOF#33

Merged
alecthomas merged 1 commit intomainfrom
aat/git-large-repo-fix
Jan 15, 2026
Merged

fix: cloning large repo's failed with fatal: early EOF#33
alecthomas merged 1 commit intomainfrom
aat/git-large-repo-fix

Conversation

@alecthomas
Copy link
Copy Markdown
Collaborator

This only manifested with --bare --mirror because the clones are so large, due to including all refs.

Also switched to using httputil.ReverseProxy, which is more robust.

~/dev/cachew $ rm -rf git-source ; time git clone --bare --mirror http://127.0.0.1:8080/git/github.com/git/git.git git-source
Cloning into bare repository 'git-source'...
remote: Enumerating objects: 807474, done.
remote: Counting objects: 100% (8533/8533), done.
remote: Compressing objects: 100% (7947/7947), done.
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
error: 297 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
git clone --bare --mirror http://127.0.0.1:8080/git/github.com/git/git.git   5.26s user 1.17s system 21% cpu 30.509 total

Once the cache is populated, cloning git takes half the time:

~/dev/cachew $ rm -rf git-source ; time git clone http://127.0.0.1:8080/git/github.com/git/git.git git-source
Cloning into 'git-source'...
remote: Enumerating objects: 403536, done.
remote: Counting objects: 100% (756/756), done.
remote: Compressing objects: 100% (363/363), done.
remote: Total 403536 (delta 532), reused 498 (delta 393), pack-reused 402780 (from 4)
Receiving objects: 100% (403536/403536), 282.29 MiB | 19.06 MiB/s, done.
Resolving deltas: 100% (305003/305003), done.
git clone http://127.0.0.1:8080/git/github.com/git/git.git git-source  18.61s user 2.65s system 96% cpu 21.940 total
~/dev/cachew $ rm -rf git-source ; time git clone http://127.0.0.1:8080/git/github.com/git/git.git git-source
Cloning into 'git-source'...
remote: Enumerating objects: 403536, done.
remote: Counting objects: 100% (403536/403536), done.
remote: Compressing objects: 100% (95986/95986), done.
remote: Total 403536 (delta 305001), reused 403485 (delta 304954), pack-reused 0 (from 0)
Receiving objects: 100% (403536/403536), 282.15 MiB | 117.21 MiB/s, done.
Resolving deltas: 100% (305001/305001), done.
git clone http://127.0.0.1:8080/git/github.com/git/git.git git-source  18.11s user 2.34s system 206% cpu 9.916 total

This only manifested with `--bare --mirror` because the clones are so
large, due to including all refs.

Also switched to using `httputil.ReverseProxy`, which is more robust.

```
~/dev/cachew $ rm -rf git-source ; time git clone --bare --mirror http://127.0.0.1:8080/git/github.com/git/git.git git-source
Cloning into bare repository 'git-source'...
remote: Enumerating objects: 807474, done.
remote: Counting objects: 100% (8533/8533), done.
remote: Compressing objects: 100% (7947/7947), done.
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
error: 297 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
git clone --bare --mirror http://127.0.0.1:8080/git/github.com/git/git.git   5.26s user 1.17s system 21% cpu 30.509 total
```

Once the cache is populated, cloning git takes half the time:

```
~/dev/cachew $ rm -rf git-source ; time git clone http://127.0.0.1:8080/git/github.com/git/git.git git-source
Cloning into 'git-source'...
remote: Enumerating objects: 403536, done.
remote: Counting objects: 100% (756/756), done.
remote: Compressing objects: 100% (363/363), done.
remote: Total 403536 (delta 532), reused 498 (delta 393), pack-reused 402780 (from 4)
Receiving objects: 100% (403536/403536), 282.29 MiB | 19.06 MiB/s, done.
Resolving deltas: 100% (305003/305003), done.
git clone http://127.0.0.1:8080/git/github.com/git/git.git git-source  18.61s user 2.65s system 96% cpu 21.940 total
~/dev/cachew $ rm -rf git-source ; time git clone http://127.0.0.1:8080/git/github.com/git/git.git git-source
Cloning into 'git-source'...
remote: Enumerating objects: 403536, done.
remote: Counting objects: 100% (403536/403536), done.
remote: Compressing objects: 100% (95986/95986), done.
remote: Total 403536 (delta 305001), reused 403485 (delta 304954), pack-reused 0 (from 0)
Receiving objects: 100% (403536/403536), 282.15 MiB | 117.21 MiB/s, done.
Resolving deltas: 100% (305001/305001), done.
git clone http://127.0.0.1:8080/git/github.com/git/git.git git-source  18.11s user 2.34s system 206% cpu 9.916 total
```
@alecthomas alecthomas requested a review from js-murph as a code owner January 15, 2026 10:07
@alecthomas alecthomas merged commit f94d404 into main Jan 15, 2026
5 of 6 checks passed
@alecthomas alecthomas deleted the aat/git-large-repo-fix branch January 15, 2026 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant