-
Notifications
You must be signed in to change notification settings - Fork 933
Add Range-based Multipart download Subscriber for Pre-signed URLs #6331
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Range-based Multipart download Subscriber for Pre-signed URLs #6331
Conversation
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
...ware/amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloadTestUtil.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
...dk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriberWiremockTest.java
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
...dk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriberWiremockTest.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces a new implementation for multipart downloads using presigned URLs with HTTP range headers instead of S3's native partNumber-based multipart API. The implementation uses reactive streams to handle concurrent part downloads while maintaining thread safety and proper response validation.
- Adds
PresignedUrlMultipartDownloaderSubscriber
class for range-based multipart downloads with presigned URLs - Implements size discovery through initial range requests and ETag validation for response consistency
- Provides comprehensive test coverage including TCK compliance tests and WireMock integration tests
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
File | Description |
---|---|
PresignedUrlMultipartDownloaderSubscriber.java | Core subscriber implementation for presigned URL multipart downloads using HTTP range requests |
PresignedUrlMultipartDownloaderSubscriberTckTest.java | Reactive Streams TCK compliance test with mocked S3AsyncClient |
PresignedUrlMultipartDownloaderSubscriberWiremockTest.java | Integration tests using WireMock for various download scenarios |
PresignedUrlMultipartDownloadTestUtil.java | Test utility class for WireMock stubbing and verification |
Comments suppressed due to low confidence (2)
services/s3/src/test/java/software/amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloadTestUtil.java:138
- The XML tag '' appears to be incomplete or truncated. It should likely be '' to match the pattern used in line 125.
.withBody("<e><Code>400</Code><Message>test error message</Message></e>")));
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.
e9d282d
to
c12ea47
Compare
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Show resolved
Hide resolved
...dk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriberWiremockTest.java
Show resolved
Hide resolved
...dk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriberWiremockTest.java
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
...dk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriberWiremockTest.java
Outdated
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Show resolved
Hide resolved
.../amazon/awssdk/services/s3/internal/multipart/PresignedUrlMultipartDownloaderSubscriber.java
Outdated
Show resolved
Hide resolved
|
…ultipartSubscriber
Motivation and Context
This PR introduces support for multipart downloads using presigned URLs. Unlike the existing
MultipartDownloaderSubscriber
which relies on partNumber, this implementation uses HTTP range headers to achieve multipart functionality with pre-signed URLs.Modifications
PresignedUrlMultipartDownloaderSubscriber
: New reactive subscriber that handles size discovery through initial range requests, manages multipart state with thread-safe operations, and validates response consistency using ETag comparison.Testing
PresignedUrlMultipartDownloaderSubscriberTckTest
: Reactive Streams specification compliance test usingSubscriberWhiteboxVerification
with mocked S3AsyncClient for isolated testing.PresignedUrlMultipartDownloaderSubscriberWiremockTest
: WireMock tests covering multipart downloads with various part sizes, single-part downloads, and error scenarios.PresignedUrlMultipartDownloadTestUtil
: WireMock stubbing utilities for range request simulation and verification.Screenshots (if appropriate)
Types of changes
Checklist
mvn install
succeedsscripts/new-change
script and following the instructions. Commit the new file created by the script in.changes/next-release
with your changes.License