diff --git a/adr/20251024-shadow-jar-dependency-bundling.md b/adr/20251024-shadow-jar-dependency-bundling.md new file mode 100644 index 0000000..93b473c --- /dev/null +++ b/adr/20251024-shadow-jar-dependency-bundling.md @@ -0,0 +1,185 @@ +# ADR: Shadow JAR for Dependency Bundling + +## Context + +The `nextflow-plugin-gradle` plugin depends on `io.seqera:npr-api:0.15.0`, which is published to Seqera's private Maven repository (`https://s3-eu-west-1.amazonaws.com/maven.seqera.io/releases`), not Maven Central or the Gradle Plugin Portal. + +When plugin projects apply `nextflow-plugin-gradle` via `includeBuild` or from a published artifact, Gradle attempts to resolve `npr-api` from the default plugin repositories, causing build failures: + +``` +Could not find io.seqera:npr-api:0.15.0. +Searched in the following locations: + - https://plugins.gradle.org/m2/io/seqera/npr-api/0.15.0/npr-api-0.15.0.pom +``` + +**Previous workaround**: Required consumers to manually add Seqera's Maven repository to their `pluginManagement` block, which is not user-friendly and error-prone. + +## Implementation + +### Shadow JAR Configuration + +**Plugin Applied**: `com.gradleup.shadow` version `9.0.0-beta6` + +**Dependency Strategy**: Use `compileOnly` for all bundled dependencies + +```groovy +dependencies { + compileOnly 'commons-io:commons-io:2.18.0' + compileOnly 'com.github.zafarkhaja:java-semver:0.10.2' + compileOnly 'com.google.code.gson:gson:2.10.1' + compileOnly 'io.seqera:npr-api:0.15.0' + compileOnly 'com.fasterxml.jackson.core:jackson-databind:2.18.2' + compileOnly 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.18.2' +} +``` + +**Rationale**: `compileOnly` dependencies are: +- Available at compile time for the plugin code +- NOT exposed to consumers via Gradle metadata +- NOT added to POM/module dependencies +- Still bundled by Shadow plugin when explicitly configured + +### Shadow JAR Task Configuration + +```groovy +shadowJar { + archiveClassifier = '' + configurations = [project.configurations.compileClasspath] + + // Relocate dependencies to avoid conflicts + relocate 'com.google.gson', 'io.nextflow.shadow.gson' + relocate 'com.fasterxml.jackson', 'io.nextflow.shadow.jackson' + relocate 'org.apache.commons.io', 'io.nextflow.shadow.commons.io' + + // Exclude Groovy - provided by Gradle + exclude 'org/codehaus/groovy/**' + exclude 'groovy/**' +} +``` + +**Key Configuration**: +- `archiveClassifier = ''` - Produces main artifact (not `-all` suffix) +- `configurations = [compileClasspath]` - Includes compileOnly dependencies +- Package relocation prevents classpath conflicts with consumer projects +- Groovy excluded as it's provided by Gradle runtime + +### JAR Replacement Strategy + +```groovy +jar { + enabled = false + dependsOn(shadowJar) +} +assemble.dependsOn(shadowJar) +``` + +**Why disable standard jar**: +- Gradle Plugin Development expects `-main.jar` for `includeBuild` +- Shadow JAR becomes the primary artifact +- Only one JAR is published/used + +### Component Metadata Configuration + +```groovy +components.java.withVariantsFromConfiguration(configurations.shadowRuntimeElements) { + skip() +} + +afterEvaluate { + configurations.runtimeElements.outgoing.artifacts.clear() + configurations.runtimeElements.outgoing.artifact(shadowJar) + configurations.apiElements.outgoing.artifacts.clear() + configurations.apiElements.outgoing.artifact(shadowJar) +} +``` + +**Purpose**: Replace default JAR artifacts with Shadow JAR in Gradle's component metadata for both runtime and API variants. + +### Test Configuration + +```groovy +dependencies { + testRuntimeOnly 'commons-io:commons-io:2.18.0' + testRuntimeOnly 'com.github.zafarkhaja:java-semver:0.10.2' + // ... (all bundled dependencies) +} +``` + +**Why needed**: Tests run in isolation and need actual dependencies at runtime, not just the shadow JAR. + +## Technical Facts + +### Artifact Characteristics + +**Size**: 8.4 MB (includes all dependencies) +- Base plugin classes: ~80 KB +- Bundled dependencies: ~8.3 MB + +**Contents verification**: +```bash +$ unzip -l build/libs/nextflow-plugin-gradle-1.0.0-beta.10.jar | grep "io/seqera/npr" | wc -l +36 +``` + +**Package relocation**: +- Original: `com.google.gson.*` +- Relocated: `io.nextflow.shadow.gson.*` +- Original: `io.seqera.npr.*` +- Kept: `io.seqera.npr.*` (no relocation, internal API) + +### Gradle Metadata + +**POM dependencies**: None (compileOnly not published) +**Module metadata**: Shadow JAR in runtime/API variants, no transitive dependencies + +### includeBuild Compatibility + +Works with `pluginManagement { includeBuild '../nextflow-plugin-gradle' }`: +- No additional repository configuration required +- Shadow JAR available on plugin classpath +- All dependencies self-contained + +## Decision + +Use Shadow JAR with `compileOnly` dependencies and package relocation to create a self-contained plugin artifact that: +1. Bundles all dependencies (including `npr-api` from private repositories) +2. Does not expose transitive dependencies to consumers +3. Relocates common libraries to avoid classpath conflicts +4. Works with both `includeBuild` and published artifacts + +## Consequences + +### Positive + +**Zero consumer configuration**: Plugin users don't need to configure Seqera's Maven repository or manage transitive dependencies. + +**Classpath isolation**: Package relocation prevents conflicts when consumers use different versions of Gson, Jackson, or Commons IO. + +**includeBuild support**: Development workflow using composite builds works without publishToMavenLocal. + +**Distribution simplicity**: Single JAR artifact contains everything needed to run the plugin. + +### Negative + +**Artifact size**: 8.4 MB vs ~80 KB for base plugin (105x larger) +- Acceptable for Gradle plugin distribution +- One-time download cost + +**Build complexity**: +- Shadow plugin configuration required +- Component metadata manipulation +- Duplicate dependencies in test configuration + +**Maintenance overhead**: +- Must keep relocation rules updated if new conflicting dependencies added +- Need to exclude Gradle-provided libraries (Groovy, etc.) + +**Version conflicts**: If consumers need different versions of relocated dependencies via plugin's extension points, they cannot override them (sealed in shadow JAR). + +### Alternative Considered and Rejected + +**Repository in pluginManagement**: Requires manual configuration by every consumer project - rejected for poor user experience. + +**Publish to Maven Central**: Not viable - `npr-api` is Seqera's internal library, not suitable for public repository. + +**Dependency substitution**: Would still require consumers to add repository and manage versions - doesn't solve core problem. diff --git a/build.gradle b/build.gradle index 948783f..d6bfd1a 100644 --- a/build.gradle +++ b/build.gradle @@ -10,12 +10,25 @@ version = file('VERSION').text.trim() repositories { mavenCentral() + maven { url = 'https://s3-eu-west-1.amazonaws.com/maven.seqera.io/releases' } + maven { url = 'https://s3-eu-west-1.amazonaws.com/maven.seqera.io/snapshots' } } - +// Use compileOnly for dependencies we'll shadow - they won't be exposed to consumers dependencies { - implementation 'commons-io:commons-io:2.18.0' - implementation 'com.github.zafarkhaja:java-semver:0.10.2' - implementation 'com.google.code.gson:gson:2.10.1' + compileOnly 'commons-io:commons-io:2.18.0' + compileOnly 'com.github.zafarkhaja:java-semver:0.10.2' + compileOnly 'com.google.code.gson:gson:2.10.1' + compileOnly 'io.seqera:npr-api:0.15.0' + compileOnly 'com.fasterxml.jackson.core:jackson-databind:2.18.2' + compileOnly 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.18.2' + + // Tests need these dependencies at runtime + testRuntimeOnly 'commons-io:commons-io:2.18.0' + testRuntimeOnly 'com.github.zafarkhaja:java-semver:0.10.2' + testRuntimeOnly 'com.google.code.gson:gson:2.10.1' + testRuntimeOnly 'io.seqera:npr-api:0.15.0' + testRuntimeOnly 'com.fasterxml.jackson.core:jackson-databind:2.18.2' + testRuntimeOnly 'com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.18.2' testImplementation 'org.spockframework:spock-core:2.3-groovy-4.0' testRuntimeOnly 'org.junit.platform:junit-platform-launcher' @@ -26,9 +39,28 @@ test { useJUnitPlatform() } +// Configure shadowJar to bundle compileOnly dependencies and be the main artifact shadowJar { archiveClassifier = '' + configurations = [project.configurations.compileClasspath] + + // Relocate dependencies to avoid conflicts + relocate 'com.google.gson', 'io.nextflow.shadow.gson' + relocate 'com.fasterxml.jackson', 'io.nextflow.shadow.jackson' + relocate 'org.apache.commons.io', 'io.nextflow.shadow.commons.io' + + // Exclude Groovy and other provided dependencies + exclude 'org/codehaus/groovy/**' + exclude 'groovy/**' + exclude 'groovyjarjarantlr4/**' +} + +// Make jar task produce the shadow JAR +jar { + enabled = false + dependsOn(shadowJar) } +assemble.dependsOn(shadowJar) gradlePlugin { website = 'https://github.com/nextflow-io/nextflow-plugin-gradle' @@ -44,3 +76,16 @@ gradlePlugin { } } } + +// Configure component metadata to use shadow JAR +components.java.withVariantsFromConfiguration(configurations.shadowRuntimeElements) { + skip() +} + +// Replace artifacts with shadow JAR +afterEvaluate { + configurations.runtimeElements.outgoing.artifacts.clear() + configurations.runtimeElements.outgoing.artifact(shadowJar) + configurations.apiElements.outgoing.artifacts.clear() + configurations.apiElements.outgoing.artifact(shadowJar) +} diff --git a/src/main/groovy/io/nextflow/gradle/registry/RegistryClient.groovy b/src/main/groovy/io/nextflow/gradle/registry/RegistryClient.groovy index d01e809..c3c13c2 100644 --- a/src/main/groovy/io/nextflow/gradle/registry/RegistryClient.groovy +++ b/src/main/groovy/io/nextflow/gradle/registry/RegistryClient.groovy @@ -1,9 +1,12 @@ package io.nextflow.gradle.registry -import groovy.json.JsonOutput -import groovy.json.JsonSlurper +import com.fasterxml.jackson.databind.ObjectMapper +import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule import groovy.transform.CompileStatic import groovy.util.logging.Slf4j +import io.seqera.npr.api.schema.v1.CreatePluginReleaseRequest +import io.seqera.npr.api.schema.v1.CreatePluginReleaseResponse +import io.seqera.npr.api.schema.v1.UploadPluginReleaseResponse import java.net.http.HttpClient import java.net.http.HttpRequest @@ -27,6 +30,8 @@ import java.time.Duration class RegistryClient { private final URI url private final String authToken + private final ObjectMapper objectMapper = new ObjectMapper() + .registerModule(new JavaTimeModule()) /** * Creates a new registry client. @@ -134,18 +139,18 @@ class RegistryClient { def archiveBytes = Files.readAllBytes(archive.toPath()) def checksum = computeSha512(archiveBytes) - // Build JSON request body - def requestBody = [ - id: id, - version: version, - checksum: "sha512:${checksum}", - spec: spec?.text, - provider: provider - ] - def jsonBody = JsonOutput.toJson(requestBody) + // Build request using API model with fluent API + def request = new CreatePluginReleaseRequest() + .id(id) + .version(version) + .checksum("sha512:${checksum}".toString()) + .spec(spec?.text) + .provider(provider) + + def jsonBody = objectMapper.writeValueAsString(request) def requestUri = URI.create(url.toString() + "v1/plugins/release") - def request = HttpRequest.newBuilder() + def httpRequest = HttpRequest.newBuilder() .uri(requestUri) .header("Authorization", "Bearer ${authToken}") .header("Content-Type", "application/json") @@ -154,21 +159,21 @@ class RegistryClient { .build() try { - def response = client.send(request, HttpResponse.BodyHandlers.ofString()) + def response = client.send(httpRequest, HttpResponse.BodyHandlers.ofString()) if (response.statusCode() != 200) { throw new RegistryReleaseException(getErrorMessage(response, requestUri)) } - // Parse JSON response to extract releaseId - def json = new JsonSlurper().parseText(response.body()) as Map - return json.releaseId as Long + // Parse JSON response using API model + def responseObj = objectMapper.readValue(response.body(), CreatePluginReleaseResponse) + return responseObj.getReleaseId() } catch (InterruptedException e) { Thread.currentThread().interrupt() throw new RegistryReleaseException("Plugin draft creation to ${requestUri} was interrupted: ${e.message}", e) } catch (ConnectException e) { throw new RegistryReleaseException("Unable to connect to plugin repository at ${requestUri}: Connection refused", e) - } catch (UnknownHostException | IOException e) { + } catch (IOException e) { throw new RegistryReleaseException("Unable to connect to plugin repository at ${requestUri}: ${e.message}", e) } } diff --git a/src/test/groovy/io/nextflow/gradle/registry/RegistryClientTest.groovy b/src/test/groovy/io/nextflow/gradle/registry/RegistryClientTest.groovy index ed0169d..70c134a 100644 --- a/src/test/groovy/io/nextflow/gradle/registry/RegistryClientTest.groovy +++ b/src/test/groovy/io/nextflow/gradle/registry/RegistryClientTest.groovy @@ -71,7 +71,7 @@ class RegistryClientTest extends Specification { .withRequestBody(containing("\"checksum\"")) .willReturn(aResponse() .withStatus(200) - .withBody('{"releaseId": 123, "pluginRelease": {"status": "DRAFT"}}'))) + .withBody('{"releaseId": 123, "pluginRelease": {"version": "1.0.0", "url": "https://example.com/plugin.zip", "date": "2024-01-01T00:00:00Z", "sha512sum": "abc123", "requires": ">=21.0.0", "dependsOn": [], "downloadCount": 0, "downloadGhCount": 0}}'))) // Step 2: Upload artifact (multipart) wireMockServer.stubFor(post(urlMatching("/api/v1/plugins/release/.*/upload")) @@ -79,7 +79,7 @@ class RegistryClientTest extends Specification { .withRequestBody(containing("payload")) .willReturn(aResponse() .withStatus(200) - .withBody('{"pluginRelease": {"status": "PUBLISHED"}}'))) + .withBody('{"pluginRelease": {"version": "1.0.0", "url": "https://example.com/plugin.zip", "date": "2024-01-01T00:00:00Z", "sha512sum": "abc123", "requires": ">=21.0.0", "dependsOn": [], "downloadCount": 0, "downloadGhCount": 0}}'))) when: client.release("test-plugin", "1.0.0", pluginSpec, pluginArchive, "seqera.io") @@ -174,7 +174,7 @@ class RegistryClientTest extends Specification { wireMockServer.stubFor(post(urlEqualTo("/api/v1/plugins/release")) .willReturn(aResponse() .withStatus(200) - .withBody('{"releaseId": 456}'))) + .withBody('{"releaseId": 456, "pluginRelease": {"version": "2.1.0", "url": "https://example.com/plugin.zip", "date": "2024-01-01T00:00:00Z", "sha512sum": "abc123", "requires": ">=21.0.0", "dependsOn": [], "downloadCount": 0, "downloadGhCount": 0}}'))) // Step 2: Upload artifact (multipart) wireMockServer.stubFor(post(urlMatching("/api/v1/plugins/release/.*/upload"))