KBFS file size limit of 512 KiB breaks git-remote-keybase

[telotortium]

While pushing my repository today I discovered the following issue:

$ git push
Initializing Keybase... done.
Syncing with Keybase... done.
To keybase://private/telotortium/repo
 ! [remote rejected] master -> master (Directory telotortium/.kbfs_git/repo/objects/pack would have increased to at least 524522 bytes, which is over the supported limit of 524288 bytes)

I tried to fix this by running git repack --max-pack-size=512k -a, but Git warns that the minimum pack file size is 1 MiB.

[strib]

That limit is on the directory size (filenames + metadata), not on individual files, so a repack should work fine. (There's also a keybase git gc --force command that might do the trick.)

Annoying that you hit that though, I've never seen git do that before. It's on our list of things to improve, maybe we should prioritize it soon.

[telotortium]

Hmm, I ran that command and now am getting on push "git-remote-keybase error: (1) malformed packed-ref". So I went to my repo, and ran git status, getting

fatal: unexpected line in home-org/packed-refs: ref: refs/remotes/origin/master refs/remotes/origin/HEAD

And sure enough, there's a line like that in HEAD. I don't know how it got there, but I've removed it for now and I'm trying the push again.

[telotortium]

When I attempt to push again, it seems to send a bunch of notifications:

2018-07-20T16:17:50.730150-07:00 ▶ [DEBU git keybase_service_base.go:1087] 6122 Sending notification for /keybase/private/telotortium/.kbfs_git/repo/objects/pack/pack-HASH.idx

After several minutes of that, I see the original error in the logs.

[strib]

see the original error in the logs.

The original error being the size limit one? Hrmm.

Are you just trying to push a single commit? Or are you trying to push a complete repo with a bunch of branches, etc in it already?

A keybase log send might help us look at it (though probably not until Monday).

[telotortium]

Log ID: 5d364409b51a5bb3c7441a1c.

[strib]

The logs don't really show anything interesting besides the error you already mentioned.

What does ls /keybase/private/telotortium/.kbfs_git/<REPO_NAME>/objects/pack show?

If it shows more than one file, maybe you can try keybase git gc --force <REPO_NAME> and then check the pack directory again to see if it reduced the number of packs at all?

(Substitute <REPO_NAME> above with the actual repo name.)

[telotortium]

It doesn't seem to have reduced the number of packs at all.

[telotortium]

I was able to work around this by deleting and re-creating the Keybase repo, running git repack -a in my local repo to reduce the number of pack files, and then running git push --force to the Keybase repo. I wonder if you could add repacking to the keybase git tool?

[strib]

Cool, glad you were able to workaround it. We actually do have re-packing implemented, but it's turned off right now:

kbfs/libgit/rpc.go

Line 279 in 6a4d2c2

MaxObjectPacks: -1, // Turn off re-packing for now

@taruti I think we had this turned off because you wanted to test it more. Do you think we can enable it now?