`$GITHUB_TOKEN` should be consumed for `ADD https://raw.githubusercontent.com/...` and `ADD https://github.com/...`

[AkihiroSuda]

Description

GitHub recently introduced the API rate limit to non-REST requests too

https://github.blog/changelog/2025-05-08-updated-rate-limits-for-unauthenticated-requests/

These changes will apply to operations like cloning repositories over HTTPS, anonymously interacting with our REST APIs, and downloading files from raw.githubusercontent.com.

source/http and source/git should be able to fetch the $GITHUB_TOKEN secret from the client so as to relax the rate limit.

Probably the existing secret API can be reused.
However, if we want to let buildx to automatically expose the token, probably it needs a new API that can be only accessed from SourceOp, not from ExecOp, to reduce security concerns.
(So, probably buildx shouldn't expose the token by default?)

[tonistiigi]

Doesn't this already work? The default secrets https://github.com/moby/buildkit/blob/v0.22.0-rc2/client/llb/source.go#L234-L235 should be added automatically in llb. Need to be either that secret value directly of with the host suffix https://github.com/moby/buildkit/blob/v0.22.0-rc2/source/git/source.go#L238-L249

[AkihiroSuda]

Doesn't this already work? The default secrets https://github.com/moby/buildkit/blob/v0.22.0-rc2/client/llb/source.go#L234-L235 should be added automatically in llb. Need to be either that secret value directly of with the host suffix https://github.com/moby/buildkit/blob/v0.22.0-rc2/source/git/source.go#L238-L249

This still doesn't work for ADD https://raw.githubusercontent.com

[AkihiroSuda]

BTW https://docs.docker.com/build/building/secrets/ should cover GitHub as well as GitLab

[tonistiigi]

This still doesn't work for ADD https://raw.githubusercontent.com

We have AttrHTTPAuthHeaderSecret now. The question is how to allow it to be set in Dockerfile. We could hardcode secret, smth like HTTP_AUTH_HEADER_<HOST>. Or invent some convention like ADD https://user:[secret=FOO]@host/path. The first one already matches what we are doing for Git, so I guess it is less controversial.

[tonistiigi]

BTW https://docs.docker.com/build/building/secrets/ should cover GitHub as well as GitLab

What's the difference?

[tonistiigi]

Are we ok with the overhead of checking HTTP_AUTH_HEADER_<HOST>, HTTP_AUTH_TOKEN_<HOST> secret on session for every URL? Or do we need to add something in the Dockerfile? ADD --auth-secret=... https:// ? As mentioned above, we already check secrets for Git, so I guess it would be ok for HTTP as well? If we do add a flag we could also make Git URLs use the same flag, although we still need to keep the default as well.

cc @marxarelli

[AkihiroSuda]

BTW https://docs.docker.com/build/building/secrets/ should cover GitHub as well as GitLab

What's the difference?

The name is the only difference I see.
It still has to be documented with a concrete example that uses GITHUB_TOKEN.
Probably a concrete GHA yaml should be listed too

HTTP_AUTH_HEADER_

SGTM in the BuildKit level, but buildx should provide a human-friendly wrapper for GITHUB_TOKEN.

[marxarelli]

Are we ok with the overhead of checking HTTP_AUTH_HEADER_<HOST>, HTTP_AUTH_TOKEN_<HOST> secret on session for every URL? Or do we need to add something in the Dockerfile? ADD --auth-secret=... https:// ? As mentioned above, we already check secrets for Git, so I guess it would be ok for HTTP as well? If we do add a flag we could also make Git URLs use the same flag, although we still need to keep the default as well.

cc @marxarelli

Thanks for looping me in!

I quite like the explicit --auth-secret flag; it seems more flexible and clear. That said, I think the conventional HTTP_AUTH_HEADER_<HOST> approach works too.