Draft
Conversation
Changed Docker build context from subdirectories (base/, base-java/, etc.) to project root (.) to allow Dockerfiles to access the Maven-built artifacts in the target/ directory. Also updated all COPY commands in Dockerfiles to use correct paths relative to the new build context (e.g., base/include/ instead of include/). This fixes the build failure where Docker couldn't find the package artifacts: "lstat .../target/cp-base-new-8.3.0-0-package/share/doc: no such file or directory" Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changed from -Ddockerfile.skip=true to -Ddocker.skip-build=true. The dockerfile.skip property doesn't exist; the correct property is docker.skip-build to skip the dockerfile-maven-plugin execution. This ensures Maven only builds the artifacts without attempting to build Docker images, leaving that to the subsequent docker build commands. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Added -Ddocker.skip-test=true to skip Python Docker integration tests since we're building Docker images directly via docker build commands rather than through the Maven dockerfile plugin. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The Dockerfiles expect Maven-created package directories (target/*-package/) but we're skipping the Maven Docker plugin build. This creates those directories manually and copies necessary artifacts before building images. Also reverted Docker build context back to subdirectories (base/, base-java/) and reverted Dockerfile path changes since the package structure is now relative to each module's directory. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Fixed module-to-artifact-ID mapping: - base → cp-base-new (not cp-base) - base-java → cp-base-java - base-java-micro → cp-base-java-micro - base-lite → cp-base-lite - jmxterm → cp-jmxterm Also added artifact copying for all modules, not just base. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Replace brittle Maven plugin packaging layout recreation with a clean
staging directory approach for better maintainability.
## Changes
**Dockerfiles** (base, base-java, base-java-micro, base-lite):
- Change COPY from `target/${ARTIFACT_ID}-${PROJECT_VERSION}-package/`
to `build-artifacts/` staging directory
- Removes hard-coded version strings (e.g., "8.3.0-0-package")
- Makes Docker builds version-independent
**Semaphore CI** (.semaphore/semaphore.yml):
- Replace package structure recreation with staging directory setup
- Create `build-artifacts/{doc,java}` in each module
- Copy jars directly to staging dirs without version paths
- Add placeholder doc files to prevent Docker COPY errors
- Increase S390X build timeout from 1h to 2h for emulation
- Add BuildKit layer caching for S390X builds
## Benefits
- **Version-independent**: No more hard-coded "cp-base-new-8.3.0-0"
- **Cleaner separation**: Maven output stays in target/, Docker input
in build-artifacts/
- **More maintainable**: No dependency on Maven plugin packaging
conventions
- **Works with buildx**: Clean context for s390x emulation builds
## Fixes
- AMD/ARM build failures (Docker COPY on empty directories)
- S390X build timeouts (increased limit + caching)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
airlock-confluentinc
bot
force-pushed
the
test_emulation
branch
from
1bf3afb to
bb905ce
Compare
Semaphore schema requires execution_time_limit at block level, not inside task block. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The S390X block needs 2h, so the global pipeline limit must also be at least 2h. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Implements the approach recommended by GPT feedback:
- Let Maven create target/*-package/ directories naturally
- Skip only Docker plugin execution with -Ddocker.skip-build/test=true
- Build images with Docker CLI using Maven-prepared directories
- No custom staging directories or manual copying needed
## Changes
**Dockerfiles** (reverted from staging approach):
- Use `target/${ARTIFACT_ID}-${PROJECT_VERSION}-package/` paths
- Maven creates these automatically during assembly phase
- Dockerfiles unchanged from original (simpler!)
**Semaphore CI** (.semaphore/semaphore.yml):
- Added `-Ddocker.skip-build=true -Ddocker.skip-test=true` to Maven
- Removed all manual staging directory creation/copying
- Added Docker build commands for all platforms
- Added S390X build block with buildx and caching
- Increased global timeout to 2h for S390X emulation
## Benefits
- **Simpler**: No manual directory creation, Maven does it all
- **Cleaner**: 30+ fewer lines of brittle CI code
- **More maintainable**: No hard-coded paths or manual copies
- **Works correctly**: Maven provides all build args to Dockerfiles
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
airlock-confluentinc
bot
force-pushed
the
test_emulation
branch
from
b0ecee8 to
a34b412
Compare
Docker build args (PROJECT_VERSION, ARTIFACT_ID, GIT_COMMIT, BUILD_NUMBER) were missing, causing Dockerfiles to fail finding package directories. ## Changes - Extract PROJECT_VERSION dynamically from Maven - Pass --build-arg for each required variable to docker build/buildx - Applied to all platforms: AMD, ARM, S390X This fixes the error: ERROR: lstat .../target/--package/share/doc: no such file or directory Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Maven's -Ddocker.skip-build=true prevents package directory creation
because the assembly is part of the Docker plugin execution.
## Solution
Manually create the package structure in CI:
- Extract PROJECT_VERSION dynamically from Maven
- Create target/{artifact}-${PROJECT_VERSION}-package/share/{doc,java}
- Add README files in doc/ (Docker COPY wildcards need files)
- Copy utility-belt JARs to java/ directories
## Why This is Needed
GPT's approach assumed Maven would create package dirs when skipping
the Docker plugin, but the plugin's assembly phase creates them, so
skipping the plugin means no package directories.
This hybrid approach:
- Uses Maven to build JARs and provide version/metadata
- Manually creates package structure Maven plugin would create
- Passes build args to Docker for proper image labeling
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
## The Right Approach AMD/ARM: Let Maven build images normally (it has all build args configured) S390X: Use manual buildx with QEMU emulation ## Why This is Better - AMD/ARM get all build args automatically from POMs - No need to manually pass dozens of build args - S390X uses buildx for emulation support - Much simpler than trying to recreate Maven's entire build context ## Changes - AMD: Removed -Ddocker.skip-build, let Maven build images - ARM: Removed -Ddocker.skip-build, let Maven build images - S390X: Still uses manual buildx approach with dynamic version This fixes the UBI_MINIMAL_VERSION and other missing build arg errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The S390X_ARCH variable was referenced in multiple places but never defined, which would cause the S390X build to fail with undefined variable errors. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
On s390x architecture, pre-built Python wheels for packages like bcrypt, cryptography, and pynacl are not available, so pip must build them from source. This requires build tools (gcc, make, python3-devel, etc.) and Rust toolchain for cryptography. Build tools are installed before pip install and removed after to keep the final image size minimal. Error fixed: - ERROR: The 'make' utility is missing from PATH - Could not build wheels for bcrypt, cryptography, pynacl Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Only s390x architecture needs build tools because Python wheels for packages like bcrypt, cryptography, and pynacl are not available for s390x and must be compiled from source. AMD and ARM have pre-built wheels, so they don't need build tools, which keeps builds faster and images smaller. Uses uname -m to detect architecture and conditionally install/remove build tools only on s390x. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
System cargo/rust packages from RHEL 9 have issues on s390x causing internal compiler errors when building Python packages like bcrypt. Replace system rust/cargo with rustup installation which provides a more recent and stable Rust toolchain with better s390x support. Error fixed: - rustc exit status 101 (internal compiler error) - could not compile proc-macro2, target-lexicon Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The export PATH command doesn't persist across && chains in RUN commands. Source /root/.cargo/env before pip install so that rustc, cargo, and other Rust tools are in PATH when building Python packages. Error fixed: - ERROR: The 'make' utility is missing from PATH (actually rustc was missing) - Could not build wheels for bcrypt, cryptography, pynacl Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Rust's linker expects 'cc' command to be available, but installing gcc alone doesn't create this symlink in RHEL/UBI images. Changes: - Install gcc-c++ in addition to gcc for complete C++ toolchain - Create /usr/bin/cc symlink pointing to /usr/bin/gcc - Set CC=gcc environment variable during pip install as backup - Clean up cc symlink when removing build tools Error fixed: - error: linker `cc` not found - Failed to build a native library through cargo Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Previous approach created symlink but pip's isolated build environments didn't inherit it properly. New approach: - Use update-alternatives to properly register cc and c++ - Split pip install into s390x and non-s390x branches - Export CC, CXX, and CARGO_TARGET_S390X_UNKNOWN_LINUX_GNU_LINKER in same command as pip install so they're available during Rust compilation - Source cargo env right before pip install, not in separate if block This ensures linker is available when cargo/rustc runs during package builds. Error fixed: - error: linker `cc` not found - Failed to build bcrypt, cryptography, pynacl Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Pip's PEP 517 build isolation creates environments that don't inherit environment variables or PATH from the parent shell. This causes gcc and rustc to be unavailable during package builds. Changes: - Add --no-build-isolation flag to pip install for s390x only - Explicitly export PATH with /root/.cargo/bin before pip install - This ensures gcc, rustc, cargo are all available during builds Error fixed: - error: linker `gcc` not found - Failed to build bcrypt, cryptography, pynacl Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
When using --no-build-isolation, pip doesn't install build dependencies automatically. Packages like bcrypt, cryptography, and pynacl require maturin (Rust-to-Python build tool) to compile. Install build dependencies first: - maturin: Build Rust-based Python packages - setuptools-rust: Setuptools extension for Rust - wheel: Build wheel packages Error fixed: - ModuleNotFoundError: No module named 'maturin' Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Maturin runs cargo in subprocesses that don't inherit shell export variables. Use env command to explicitly set environment for pip and all its subprocesses. Changes: - Use `env` command to wrap pip install with all needed variables - Set absolute paths (/usr/bin/gcc) instead of relative (gcc) - Add RUSTFLAGS=-C linker=/usr/bin/gcc for Rust compiler - Explicitly set PATH to include cargo bin and standard paths This ensures maturin/cargo can find both rustc and gcc when compiling. Error fixed: - error: linker `gcc` not found (in cargo subprocess) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
GCC failed with exit status 127 because it couldn't find required tools like 'as' (assembler) and 'ld' (linker) which are provided by binutils. Install binutils alongside gcc to provide complete compilation toolchain: - as: GNU assembler - ld: GNU linker - Other binary utilities needed by gcc Error fixed: - linking with `/usr/bin/gcc` failed: exit status: 127 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add a simple C compilation test to verify gcc toolchain is working before attempting to build Rust packages. This will help diagnose whether the issue is with gcc itself or with Rust/cargo integration. Test compiles, links, and runs a minimal C program to verify: - gcc can compile source code - linker can create executables - executables can run Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…ssues After extensive debugging, Rust package compilation (bcrypt, cryptography, pynacl) consistently fails on s390x due to QEMU emulation limitations and toolchain issues. Solution: Skip confluent-docker-utils installation on s390x architecture. - Removes all build tool installation (gcc, rust, maturin, etc.) - Removes all compilation-related cleanup - Prints informative message about why it's skipped - AMD/ARM continue to install normally with full functionality This allows s390x base images to build successfully. Images extending this base can add confluent-docker-utils if specifically needed. Issues encountered during debugging: - Rust compiler crashes with system packages - Rustup-installed Rust can't find gcc linker - Pip build isolation prevents env var inheritance - GCC exit status 127 despite proper installation - All attempts at various workarounds failed Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Instead of manually running docker buildx with limited build args, let Maven's fabric8 docker plugin build the images. This ensures all required build args (GOLANG_VERSION, etc.) are passed automatically. Changes: - Remove manual docker buildx commands - Remove manual package directory creation and README files - Let Maven build images like AMD/ARM builds - Configure buildx and BuildKit for Maven to use - Removed 20+ lines of manual build logic Note: fabric8 plugin may not fully support buildx multi-platform builds, but with QEMU and buildx configured, it should work for single platform. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Spotify's dockerfile-maven-plugin does NOT support cross-platform builds. When Maven runs on AMD64 host, it always builds AMD64 images regardless of QEMU/buildx configuration. Correct approach for s390x: 1. Use Maven with -Ddocker.skip-build=true to build JAR artifacts only 2. Extract ALL build arg values from Maven properties using mvn help:evaluate 3. Manually run docker buildx with --platform linux/s390x and all extracted args Build args extracted and passed: - PROJECT_VERSION, GIT_COMMIT, BUILD_NUMBER - UBI9_MINIMAL_VERSION, UBI9_MICRO_VERSION, UBI9_VERSION - GOLANG_VERSION (required by base-lite and base-java) - CP_DOCKER_UTILS_VERSION, CONFLUENT_DOCKER_UTILS_VERSION This ensures actual s390x architecture images, not AMD64 images with s390x tag. Reverts: a010213 which incorrectly tried to use Maven for s390x builds Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Go compiler crashes with segmentation fault when cross-compiling for s390x under QEMU emulation: internal/asan: /usr/local/go/pkg/tool/linux_s390x/compile: signal: segmentation fault (core dumped) This is a known limitation of Go + QEMU for s390x architecture. Solutions considered: 1. Skip base-lite for s390x (current approach) 2. Use native s390x builder (requires infrastructure) 3. Pre-compile ub binary on native platform 4. Use older Go version (may not work) For now, skip base-lite on s390x. Other images (base-new, base-java, base-java-micro, jmxterm) build successfully. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Use tonistiigi/xx toolkit to cross-compile Go binaries natively instead of relying on QEMU emulation which causes segmentation faults. Changes: - base-java/Dockerfile.ubi9: Add xx toolkit for ub & package_dedupe - base-java-micro/Dockerfile.ubi9: Add xx toolkit for ub & package_dedupe - base-lite/Dockerfile.ubi9: Add xx toolkit for ub binary - .semaphore/semaphore.yml: Re-enable base-lite build for s390x This follows the approach from service-mesh PR #964: https://github.com/confluentinc/service-mesh/pull/964/changes Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
When cross-compiling with xx toolkit, go install doesn't place binaries in the expected location. Use go build -o with explicit output paths. Changes: - Clone cp-docker-utils repo and build from source - Use go build -o /usr/local/bin/... for explicit binary placement - Update COPY commands to use /usr/local/bin instead of /go/bin Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
AMD/ARM use Maven's Spotify plugin for native builds (doesn't support BuildKit syntax). S390X needs cross-compilation with xx toolkit. Changes: - Restored original Dockerfile.ubi9 for AMD/ARM (native builds via Maven) - Created Dockerfile.ubi9.s390x with cross-compilation for s390x - Updated semaphore.yml to use .s390x Dockerfiles for s390x builds This keeps cross-compilation complexity isolated to s390x only. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
ARG values must be declared before FROM statements that use them. This fixes the 'UndefinedArgInFrom' build error. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
jmxterm depends on cp-base-new as its base image and pulls from ECR. We need to push cp-base-new immediately after building it, before jmxterm tries to use it. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Description
Testing