Support alternative GCP universes #1142
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
S3NS is a new deployment of GCP, with operations delegated by Google to Thales: https://www.s3ns.io/en S3NS is officially supported by Google, but is totally separate from the regular GCP, with notably its own API domains. We're porting our platform to S3NS, which includes CNPG configured with the Barman Cloud plugin for backups, which doesn't work there right now because we can't configure the Google *universe*. To connect to S3NS, official clients need to be configured with a universe value, that overrides the base API domain. `GOOGLE_CLOUD_UNIVERSE_DOMAIN` is the official environment variable used to target S3NS: https://documentation.s3ns.fr/docs/overview/tpc-key-differences#key_differences_for_developers Unfortunately Barman uses the v1 client that doesn't read this variable, and only accepts the universe as part of the `client_options` field in the constructor, so we have to pass it in like this.
Author
|
This is blocking us from deploying on S3NS, since without backups are DBs are nothing 🙂 Happy to discuss this further. S3NS is very new (they went GA less than two months ago) but will certainly grow in usage. Note that the v2 Storage client reads the environment variable by default: https://github.com/googleapis/python-storage/blob/f7095faf0a81239894ff9d277849788b62eb6ac5/google/cloud/_storage_v2/services/storage/client.py#L464 Switching to it would of course fix the issue, but the fix in this PR should be enough to support S3NS without a big upgrade. |
mgarstecki
changed the title
Contributor
|
@mgarstecki, thanks for your PR. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In multiple countries, local hosting companies are partnering with GCP to offer EU-sovereign GCP-like environments. Such partnerships include one with T-Systems in Germany or one with Thales in France: S3NS.
For that, Google introduced the notion of
universein their SDKs and CLI tools, to basically point them at non-Google infrastructure.We're currently porting our platform to S3NS, which includes CNPG with the Barman Cloud plugin for backups, which doesn't work there right now because we can't configure the Google universe.
To connect to other universes, official clients need to be configured with a universe value, that overrides the base API domain. GOOGLE_CLOUD_UNIVERSE_DOMAIN is the official environment variable used to target non-GCP universes: https://documentation.s3ns.fr/docs/overview/tpc-key-differences#key_differences_for_developers
Unfortunately Barman uses the v1 client that doesn't read this variable, and only accepts the universe as part of the
client_optionsfield in the constructor, so we have to pass it in like this.