Improve handling of S3 offloading configuration in other regions #9084
Description
Is your enhancement request related to a problem? Please describe.
Currently, when using S3 offloading in a region other than us-east-1, the underlying jclouds library does some non-standard AWS things which require both:
A) the policy Pulsar is running as to have GetBucketLocation permissions and
B) to change the endpoint to use a region specific endpoint
This causes confusion which is not well documented and difficult to explain, and different from most AWS implementations. See #3833 for context
Describe the solution you'd like
We should do 2 things
- See if we can eliminate the need for
GetBucketLocation, looking at https://github.com/apache/jclouds/blob/31a3e5b5df1543d04098e3a694130b7ae8e6e079/apis/s3/src/main/java/org/jclouds/s3/config/S3HttpApiModule.java#L91 it appears to only be used when jclouds detects multiple regions. Where jcloud is getting more than one region from isn't clear, but if the user sets a region, we should just use that single region and skip the getBucketLocationCheck - Ensure that setting just the region is sufficient to configure the correct endpoint. Getting rid of the
GetBucketLocationcheck may be sufficient such that the default endpoint works, otherwise, we should build the correct endpoint name if the region is specified but no endpoint is manually provided
Describe alternatives you've considered
Another consideration (and perhaps still a longer term goal) is to replace the use of jcloud for AWS (but still use it for other cloud providers) as jcloud does have some other behavior that differs from AWS.
Additional context