support user-configured authorization header

[StephenWithPH]

Issue Kind

Brand new capability

Description

I believe users would benefit from a behavior along the lines of:

• poetry config http-authorization-header.foo "<authorization header content, including the scheme>"
• export POETRY_HTTP_AUTHORIZATION_HEADER_FOO="<authorization header content, including the scheme>"

... where foo is name of the repository to which http requests should have the configured header.

Adding this feature would allow poetry users to use any private package index which supports use of the Authorization header, and should free the poetry developers from feature requests and issues dealing with any given private package index's authorization.

Note that I believe it is best to leave the choice of authentication scheme up to the user; that is, the user would be expected to set the string to something like Bearer <their bearer token>. This allows poetry to support any authentication scheme without additional work.

---

Like others, I have struggled to use poetry with a private package index. Ultimately, the challenge is in satisfying the private index's authorization requirements using poetry's available configuration knobs.

In the case of JFrog, they support:

• Basic authentication using username and password
• Using an access token instead of a password for basic authentication
  • in the basic auth <username>:<password> structure, this becomes :<token>
  • I believe such form was (inadvertently?) working until poetry 2.0.0; see poetry 2.0 authentication failure #9977
  • I don't believe there is any way to get poetry >= 2.0.0 to properly authenticate with JFrog using ✨ only ✨ an access token
• Using an access token as a bearer token in an authorization header (Authorization: Bearer) with your access token

Impact

Adding this feature would allow poetry users to use any private package index which supported use of the Authorization header.

Workarounds

Whether or not poetry works today with a given private package index depends on the interaction of that index's authentication methods and poetry's available feature set.

In the case of JFrog, I do not believe there is any workaround using only an API token; there are JFrog use cases where there is no username, and these basic auth forms do not work:

• username=<token> password=<token>
• username=<empty string> password=<token>

[dimbleby]

in the basic auth username:password structure, this becomes :token

that is not what the jfrog docs say:

In this case, it is important to access Artifactory using the same user name provided when creating the token

in which case username / password continue to be a sufficient option

[StephenWithPH]

that is not what the jfrog docs say

Interesting. Empirically, poetry 1.8.5 with no username and password=<token> works when talking to JFrog. I assume that translated to basic auth :<token> in 1.8.5.

Perhaps that is related to JFrog tokens that are not user-scoped: https://jfrog.com/help/r/jfrog-platform-administration-documentation/support-authentication-for-non-existing-users

Regardless, do you think the utility of coercing the Authorization header still stands?

[dimbleby]

If there is in fact no use case for it then I would definitely be against it.

The configuration that you propose sounds rather awkward to me, I do not think that people would enjoy working with it. If there truly is a good reason for eg supporting bearer tokens then probably someone ought to think a bit more about a way of making that more palatable

[StephenWithPH]

If there is in fact no use case for it then I would definitely be against it.

My concrete use case is that I need to use a "userless" access token from JFrog with poetry. I was able to do so by (ab)using poetry's existing basic http auth machinery by providing no username and the token as password, but that seems to have broken in poetry 2.0.0.

The configuration that you propose sounds rather awkward to me, I do not think that people would enjoy working with it. If there truly is a good reason for eg supporting bearer tokens then probably someone ought to think a bit more about a way of making that more palatable

Scoping this down to supporting only bearer tokens is perfectly reasonable. I think that choice would make the usage pattern less awkward:

• poetry config http-bearer-token.foo "<token>"
• export POETRY_HTTP_BEARER_TOKEN_FOO="<token>"

Does this seem more palatable to you?

[dimbleby]

And did you try providing a username? It has been a while since I used jfrog but my recollection is that it just didn't care what the username was when providing an access token.

(Though this too seems to contradict their docs)

[StephenWithPH]

And did you try providing a username? It has been a while since I used jfrog but my recollection is that it just didn't care what the username was when providing an access token.

Wow. That was enough of a breadcrumb to induce the following. You are my hero!

With poetry 2.0.0, all of these fail:

password = "<redacted token>"

username = ""
password = "<redacted token>"

username = "ThisIsNotAValidUsername"
password = "<redacted token>"

... but this one succeeds:

username = " "
password = "<redacted token>"

I speculate that as username = " " flows through the machinery poetry uses for http basic auth, it may ultimately reduce to basic auth of the form :<token>... but that's hard to say.

---

Having said all of the above, I think directly supporting the user's ability to set a bearer token would be a very useful feature. The level of workaround to coerce a bearer token into basic auth is cumbersome problematic, in that this may be a JFrog-specific undocumented feature and/or inadvertent behavior of poetry or its dependencies... either of which may cease to work in the future.

[abn]

I think directly supporting the user's ability to set a bearer token would be a very useful feature

Question is, how useful? As far as I can tell all of the python package indices advertise document username/password authentication even when tokens are used. Granted this is largely because majority ecosystem tooling only supports this method.

As things, stand I do not see value proportional to maintenance burden in Poetry implementing an additional feature to support this. I might be missing more context here, folks can provide that if they think this is a valuable feature.

[StephenWithPH]

I did some digging into the broader Python packaging ecosystem's state regarding authentication methods other than basic auth. Upon reflection, this GitHub issue is really a reflection of a larger, known feature request that shows up in several places (I omitted several similar issues):

• PEP 717: Delegated Repository Authentication python/peps#3172
• Uploads: Remove the username=__token__ requirement? pypi/warehouse#15140
• https://github.com/psf/fundable-packaging-improvements/blob/master/FUNDABLES.md#architecture-to-support-alternative-authentication-methods-in-packaging-tools
• pip should support custom authentication handlers for private pypi pypa/pip#4475
• Allow pip to send arbitrary headers to various hosts pypa/pip#8042

Those proposals run the gamut from "just support auth headers" to "delegate it all to pluggable magic that can handle many different auth methods".

This appears to be a standardization / coordination / collective action problem; it likely falls to Python Packaging Authority (PyPA) to address. Notably, their current specification for package index interfaces blesses (ab)using the basic auth format to transmit a token.

In the very long run (Python 4 😆 ), I think support for bearer tokens will become part of the official standard and be supported by PyPI as an auth method. But until there is some standardization, I sympathize with the Poetry project's need to balance maintenance considerations, the risk of running ahead of future standardization, and implicitly supporting various private PyPI vendors' implementations.

You're welcome to close this issue as unplanned, or let me know and I will do so.

If upon reviewing all of this, you're open to supporting bearer tokens, I'm happy to put together a list of the private PyPI vendors for whom I believe that would work.

[abn]

@StephenWithPH appreciate you doing the research there.

If you do not mind doing the research, I would love to see the list of vendors providing private indices. In particular, those for which use of a bearer token (or auth header) would clearly improve the ease of use for Poetry users.

For the empty username scenario that you mention in this issue, the auth header is more an escape hatch.

Before we decide, I would definitely like to see the list of vendors providing private index features that would benefit from this. As in, are there enough cases where it might make it worthwhile for Poetry users making it more convenient.

Also, I should note that with keyring support, you can also do things like this to get credentials for things like google etc.

pipx inject poetry keyrings.google-artifactregistry-auth

[StephenWithPH]

@abn ... having done yet more digging, all of the self-proclaimed universal artifact repositories seem to follow a similar pattern.

First, they support some form of key or token across their entire API, using http headers as one transmission mechanism:

• https://help.cloudsmith.io/reference/authentication#api-key-authentication
• https://docs.inedo.com/docs/proget/reference-api/proget-apikeys#using-api-keys
• https://jfrog.com/help/r/jfrog-platform-administration-documentation/authorization-headers

In addition, they have package-format-specific authentication documentation (using language-specific clis). For Python, it makes sense that all three must salute the de-facto PyPI standard of basic auth, since they all must work with pip and its peers.

They do so for keys/tokens, sometimes replacing username with a magic string:

• https://help.cloudsmith.io/docs/python-repository#setup
  • select the HTTP Basic Auth (Token) tab, and see that they use the literal token as username
• https://docs.inedo.com/docs/proget/feeds/pypi#authenticating-to-a-pypi-feed
  • they use the literal api as username
• https://jfrog.com/help/r/jfrog-artifactory-documentation/set-up-pypi-clients-to-work-with-artifactory
  • these Python-specific docs don't speak to non-user-scoped tokens at all, but refer back to "userless" access token and support user-configured authorization header #10037 (comment)

With time and access to the other platforms, I suspect one could figure out how to use each platform's "generic" auth patterns for Python.

But I do further appreciate the question of whether or not poetry should try to work around the Python-ecosystem-wide challenge of better-than-basic-auth, especially given the requirement to have access to each platform in order to verify that any given solution would work.

At your leisure, let me know what you think.