Open
Conversation
openstack-mirroring
force-pushed
the
master
branch
from
3f3f3c4 to
479a010
Compare
openstack-mirroring
force-pushed
the
master
branch
from
dd19613 to
aa04c3e
Compare
…g requests The labeled timing metrics previously have sample_rate similar to those added for non-labeled metrics. However the down sampling has not been helpful when using labeled metrics to investigate customer issues, for example those related to object server REPLICATE requests. This patch changes labeled timing metrics to not have sample_rate. The non-labeled metrics are unrelated to this effort thus not changed. Related-Change: I05336b700120ab5fcf922590d6a12f73112edb50 Change-Id: Ia6e856ffaf8fd1b4a905e6976ebdc62ed5ddf32f
…ebugging requests"
Swift does not return all the parameters of objects in a listing (e.g. ChecksumType and ChecksumAlgorithm) so pop these from listings before making assertions. Change-Id: Ieb7a9783731c11f1c08db398eae07ffafa127460
Change-Id: Iea1adfb93891e4bde62a11bfcef478d9b9696fd4
It used to be that when a logger encountered an error trying to handle / emit a message, it would attempt to log about the error via the same logger. Long ago, this could lead to infinite recursion in Swift's LoggerFileObject, so we fixed it and added a test that included showing the bad stdlib behavior. Recently, stdlib did a similar fix so the recursion doesn't happen. See - python/cpython#91555 and - python/cpython#131812 Of course, now our test breaks (at least, on 3.13.4). Relax the assertion a little. Related-Change: Ia9ecffc88ce43616977e141498e5ee404f2c29c4 Change-Id: Id2f490d4204b8eaf07857cb84ed783bec19b8511
Change-Id: I32fec7540bee70e77140964c5983d133a572fa7b
... because Python 2.x is no longer supported. Change-Id: I3167a539b3e26ceb35976fbd7a2356ba59d4a5e4
Change-Id: I4eff901aaa334c8a73cebfc578cea14d23e6c365
datetime.timezone.utc[1] has been available in Python 3 and can be used instead of datetime.UTC which is available only in Python >=3.11 . [1] https://docs.python.org/3.13/library/datetime.html#datetime.timezone.utc Change-Id: I92bc82a1b7e2bcb947376bc4d96fc603ad7d5b6c
Change-Id: I74381567978025e7f528e169692bbaf9bac45de2
The existing doc is quite outdated and is based on ancient versions. Update it according to the following points, so that it works in recent versions. * Use python3- packages instead of python-/python2- packages * xinetd is no longer available in recent CentOS * Remove old unused test dependencies such as nose and mock * Remove netifaces which is no longer a hard dependency Change-Id: I8bf87f858406dc1a32139a0071b53cfb90864108
Change-Id: I80142c6c0654b65b5755e7e828bcc4969a10f4f1
Change-Id: I88e2e743ccbd6292bc1570ae0efbdd45dcced8cc
If we already know enough about the error to feel confident in quarantining, we don't really need a whole traceback about it; that just clutters logs. Change-Id: Iab0e62c85b33d699f96d744faaa16420b7148b47
S3 includes the expected base64 digest in a BadDigest response to a multipart complete POST request. Co-Authored-By: Tim Burke <tim.burke@gmail.com> Change-Id: Ie20ccf10846854f375c29be1b0b00b8eaacc9afa
For more information about this automatic import see: https://docs.openstack.org/i18n/latest/reviewing-translation-import.html Change-Id: I9a31eda60da71ff9f9db8b32b517338e99896667 Signed-off-by: OpenStack Proposal Bot <openstack-infra@lists.openstack.org> Generated-By: openstack/openstack-zuul-jobs:roles/prepare-zanata-client/files/common_translation_update.sh
See https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html for some background. This covers both "normal" objects and part-uploads for MPUs. Note that because we don't write down any client-provided checksums during initiate-MPU calls, we can't do any verification during complete-MPU calls. crc64nvme checksums are not yet supported; clients attempting to use them will get back 501s. Adds crt as a boto3 extra to test-requirements. The extra lib provides crc32c and crc64nvme checksum support in boto3. Co-Authored-By: Ashwin Nair <ashnair@nvidia.com> Co-Authored-By: Alistair Coles <alistairncoles@gmail.com> Signed-off-by: Tim Burke <tim.burke@gmail.com> Signed-off-by: Alistair Coles <alistairncoles@gmail.com> Change-Id: Id39fd71bc59875a5b88d1d012542136acf880019
Adds a test that verifies extra body content beyond the content-length is ignored provided that the checksum value matches that of the content-length bytes. Add comment to explain why this is the case. Drive-by: add clarifying comment to unit test. Change-Id: I8f198298a817be47223e2f45fbc48a6f393b3bef Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Assert that BadDigest responses due to checksum mismatch do not include the expected or computed values. Change-Id: Iaffa02c3c02fa3bc6922f51ecf28a39f4b24ccf2 Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Previously, test_reclaim_tmp_files assumed a particular order to the
result of os.listdir. However, no order is guaranteed from python
os.listdir and the test would fail on some platforms (including
macos).
[1] "The list is in arbitrary order"
https://docs.python.org/3/library/os.html#os.listdir
Change-Id: If26c961c2133ec48c32a29e37a4a427aa8a4c818
Related-Change: If30005269d40a1a3f711008b5d3b863efc9fb683
Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
The py3 tag is preserved with multiple tags in the swift-*-image jobs. Signed-off-by: Tim Burke <tim.burke@gmail.com> Change-Id: Ic1a5f5ed4ab5fe0fd278083a919efabf39f72c56
The author has observed this test fail because the request somehow succeeded in connecting to 10.0.0.x addresses, meaning that node timings *were* unexpectedly set. Even when the test behaves 'normally' there is a >= 0.5 sec delay waiting for a connection timeout. This is unnecessary: we can simply mock the connections to raise Timeouts immediately. Change-Id: Iff2f0e6dffca73eb689bf354e278e0f727f881af Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Signed-off-by: Tim Burke <tim.burke@gmail.com> Change-Id: I53445f574953d9d47dd93d6819060e4b3cabee75
Signed-off-by: Matthew Oliver <matt@oliver.net.au> Change-Id: Ieaf9ca9a67e40a7133f56f4fde86a88ba4666ac4
Signed-off-by: Tim Burke <tim.burke@gmail.com> Change-Id: I202ac65afd0fa1812cc7b5e0b6775489ac1daedd
Signed-off-by: Tim Burke <tim.burke@gmail.com> Change-Id: I5ad18f1ef0ef040fc65d34904fad2d324b451d0d
This commit adds support for barbican_region_name configuration option in Swift's kms_keymaster middleware to enable proper multi-region Barbican endpoint selection via Castellan. Changes: - Added 'barbican_region_name' to keymaster_opts tuple to allow reading the option from configuration - Set barbican_region_name in Castellan's config after calling options.set_defaults() so it's available to BarbicanKeyManager - This enables Castellan's BarbicanKeyManager to use the region for endpoint discovery in multi-region deployments The barbican_region_name option should be set in the [kms_keymaster] section of the keymaster configuration file (or proxy-server.conf). Castellan's BarbicanKeyManager will use this value when discovering the Barbican endpoint from the service catalog. Related-Bug: #2138973 Signed-off-by: Ade Lee <alee@redhat.com> Change-Id: I02e86c736398698d1bae19c17e9d97b1974e5054
Signed-off-by: Tim Burke <tim.burke@gmail.com> Change-Id: I21c13b42f5d35b0ac14d71f3cc115bac12503aec
There's no need for the s3api controllers to set the x-timestamp header in requests. Furthermore, it may be inappropriate for the s3api middleware to set the x-timestamp header if in doing so it overwrites a value set by a previous middleware. In the case of an object copy, the last-modified time required for the response body can be obtained from the swift copy (PUT) request response. The proxy server app copies the PUT request.timestamp to the PUT response.last_modified. Related-Change: Iab68f4b596dfa7f23bc587fbde859d854faff87f Related-Change: I4dacc11ec086fdd2e9bb3f68295f2ebaea68bede Change-Id: I4f46ae51579aafb9f511ddec8f2bce5fd5498ba5 Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
S3 and Swift both support expiring objects in some form, but the
semantics are entirely different.
In S3, expiration policy is at the bucket level. A client calls
PutBucketLifecycleConfiguration to specify bucket lifecycle rules;
some of these rules enable object expiration. After that API call,
the rules take effect within a few hours for all objects in the bucket.
In Swift, expiration policy is at the object level. When uploading an
object, a client can add the header “X-Delete-At: T” or
“X-Delete-After: D”, where T is a Unix timestamp and D is a duration
in seconds. That object will then expire at either time T or at
D seconds after the PUT request was made.
S3 object GET and HEAD responses for expiring objects contain the time
at which the object will expire based on the current bucket configuration
as well as the rule responsible. This is in the “X-Amz-Expiration” header.
Example:
x-amz-expiration: expiry-date="Fri, 16 Jan 2026 00:00:00 GMT",
rule-id="logs-expiration"
This patch adds this support into Swift’s S3API for expiring objects.
Swift's S3API does not currently support expiration via bucket lifecycle
policies, so the x-delete-at header is the only possible source of
expiration time to be returned in the x-amz-expiration header. However,
if bucket lifecycle expiration were to be implemented then the
x-amz-expiration header could return the lowest of the bucket expiration
time or the x-delete-at time.
Adds Sam, who came up with this approach, as co-author.
Change-Id: I9e319b75c557d5ab971182adb4aa10ad27a14d0b
Co-authored-by: Samuel Merritt <smerritt@nvidia.com>
Signed-off-by: Jianjian Huo <jhuo@nvidia.com>
Change-Id: If05f93a74f576b485118944a06a51b09f8fdd54a Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Previously, in object_versioning middleware, when writing a delete marker to the versions container, the timestamp value used to calculate the delete marker version id was different to the x-timestamp given to the delete marker PUT sub-request. This was inconsistent with the behaviour when writing a version to the versions container, where the x-timestamp value of the PUT to the versions container is the same as the value used to construct the version id. With this patch, the same timestamp value is used for the delete marker version id and the corresponding PUT sub-request. The patch also adds tests to verify the x-timestamps that are applied by the object_versioning middleware. The FakeSwift test helper class is extended to support default responses which will match any path for a given path. This allows requests to be captured whose path may not be known a priori (for example, the path to a new version object). Change-Id: I1c2794bb7ac4eb984ae6b48f41ecafa76e9e7e4c Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Previously, requests would be assigned an x-timestamp header (if they do not already have one*) by various means in the proxy account, controller and object controllers. With this patch, all requests are assigned an x-timestamp header at one place early in the proxy server Application (i.e. the update_request method). BaseController.generate_request_headers now copies an x-timestamp from the original request, rather than setting x-timestamp. Note: a request may already have been assigned an x-timestamp in middleware (e.g. SLO, object versioning). Previously, the proxy Application would only set an x-timestamp header if none existed (in BaseController.generate_request_headers), *except* for container PUT, UPDATE and DELETE requests which would have an existing x-timestamp header replaced (in ContainerController._backend_requests). With this patch the proxy Application never overwrites an existing x-timestamp header. Drive-by: The BaseController.generate_request_headers orig_req argument is no longer optional. All existing call sites pass an orig_req value. Change-Id: Ic32d32f27157d1ba47e169ff1a22bce6f5a89395 Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
It's not immediately obvious, but the direct client functions always set an x-timestamp header in request headers if the header does not already exist. This patch refactors the module so that the timestamp is set in one place. Drive-by: fix missing call to do_test helper in test_direct_get_container_with_extra_params. Change-Id: I7981525dc8d800aa9304fb5fb837e5361b5eeaa0 Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
The author found the print output confusing while trying to sanity check the output of the test's logger during review of the Related-Change. Related-Change: I046c4837a638d097649688db39455fb089fa7007 Change-Id: I81fed3bb02da33322804813636223949ec03fdc4 Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
Various parts of the relinker record the invoked action ('relink',
'cleanup') in generated log messages, without utilizing a uniform
format. In most cases, the logged output included cleanup=(True/False).
This patch improves this by using a prefixed logger that prepends log
messages with the invoked action's name, which seamlessly supports any
actions added to the relinker in the future.
Change-Id: I046c4837a638d097649688db39455fb089fa7007
Signed-off-by: Wael Halbawi <whalbawi@nvidia.com>
Jammy is not in testing runtime for this cycle and we can drop this job which is also failing 100%. Change-Id: Ia9e5782ee8f632951412898895a6134464e9dc3e Signed-off-by: Ghanshyam Maan <gmaan.os14@gmail.com>
Strengthens the assertions (previously relaxed in [1]) in TestRelinker::test_state_file. The test now asserts that the step (relink/cleanup) is included in the logged output. [1] I046c4837a638d097649688db39455fb089fa7007 Change-Id: I4449e8fded45b74db4463008c9839a33b48ffd17 Signed-off-by: Wael Halbawi <whalbawi@nvidia.com>
Previously, normalize_timestamp was widely used in tests to generate a string representation of a Timestamp. However, tests should often be using the internal format of a Timestamp. This patch retires the use of normalize_timestamp in tests. The preferred mechansim for generating sample timestamps in unit tests is to use the timestamp iterator helper available via BaseUnitTestCase.ts_iter and BaseUnitTestCase.ts(). This patch adopts this pattern, but there is still further work required to make all tests use the pattern consistently. Change-Id: I2b6cee7e7a189294d4e28b7cb2aadadd10a11c2d Signed-off-by: Alistair Coles <alistairncoles@gmail.com>
test_db was using its own self.ts which was a timestamp_iter whereas other parts of swift unittests inherit their testcase from BaseUnitTestCase which has `self.ts_iter` as the timestamp_iter and self.ts as a helper method which calls `next(self.ts_iter)`. Meaning this code although looking similar actually behaves quite differently. So this patch changes test_db to use the BaseUnitTestCase everywhere it uses `self.ts` to keep it consistent with other tests in swift. In summary, TestDbBase extends BaseUnitTestCase and next(self.ts) becomes self.ts(). Change-Id: I4db72dc72a274838cbc4cd54e8ce17770eddfd0a Signed-off-by: Matthew Oliver <matt@oliver.net.au>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
18 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.