Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions hyperfleet/adrs/0013-konflux-build-and-release.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
Status: Active
Owner: Ciaran Roche
Last Updated: 2026-04-21
---

# 0013 — Konflux for Container Image Build and Release

## Context

HyperFleet builds container images for three components (API, Sentinel, Adapter) using Prow's ci-operator, publishing to `registry.ci.openshift.org`. This setup lacks supply chain security: no signed provenance records, no SBOM generation, no image signing.

Red Hat requires all services to use Konflux for image builds to achieve SLSA Level 3 compliance. Konflux provides provenance via Tekton Chains, automatic SBOM generation, image signing with cosign, and Enterprise Contract policy enforcement — capabilities Prow does not offer.

Meanwhile, HyperFleet's E2E testing infrastructure runs on Prow with GKE clusters, GCP credentials, and an established test framework that would be costly and complex to replicate in Konflux.

## Decision

Adopt a hybrid model: **Konflux handles image build and release, Prow retains E2E testing.**

The architecture has three layers:

- **PaC (Pipelines as Code)** — trigger layer. Watches GitHub webhooks and matches git events (push to main, version tags) to `.tekton/` pipeline files in each component repo.
- **Konflux** — orchestration layer. Builds images, generates SBOM, signs provenance (Tekton Chains), runs Enterprise Contract validation, and releases to `quay.io/redhat-services-prod/hyperfleet/`.
- **Prow** — test layer. Runs nightly and RC E2E tests against Konflux-built images from Quay. Retains PR presubmit checks (lint, unit tests).

Key design choices:

- One Application (`hyperfleet`) with three Components, releasing through a single ReleasePlanAdmission with auto-release (`block-releases: false`)
- Tag-based triggers using CEL expressions (not globs) to distinguish RC tags from release tags
- `app-interface-standard` Enterprise Contract policy for service-type releases
- RC E2E triggered via `hyperfleet-release` repo tag -> GitHub Action -> Gangway -> Prow
- `rh-push-to-external-registry` release pipeline with Pyxis registration and Slack notifications

See [Konflux Release Pipeline Design](../docs/release/konflux-release-pipeline-design.md) for the full design document.

## Consequences

**Gains:**

- SLSA Level 3 supply chain security (provenance, SBOM, image signing) with zero extra configuration
- Simplified RC testing — one reusable Prow job with parameterized image coordinates replaces per-release-branch pipeline definitions
- Auto-release on merge to main eliminates manual image publishing
- Public Quay images remove pull secret complexity for Prow and partner environments
- Alignment with Red Hat release engineering standards

**Trade-offs:**

- Two build systems during transition (Konflux for images, Prow for PR validation and E2E)
- RC E2E triggering depends on Gangway API availability
- Team must learn Konflux primitives (RPA, ITS, PaC, Snapshots)
Comment thread
rh-amarin marked this conversation as resolved.

## Alternatives Considered

| Alternative | Why Rejected |
|-------------|--------------|
| Full Prow (status quo) | No path to SLSA compliance. No provenance, SBOM, or signing. Does not meet Red Hat release engineering requirements. |
| Full Konflux (including E2E) | Would require replicating GKE cluster provisioning, GCP credentials, and the entire hyperfleet-e2e framework inside Konflux ITS pipelines. High cost, low benefit — Prow already works. |
| Hybrid with ITS wrapper for E2E | An IntegrationTestScenario that triggers Prow via Gangway and polls for results. Adds complexity (polling, timeouts, error handling) as a Tekton pipeline when a simpler GitHub Action achieves the same result with better DX (retrigger via `gh workflow run`). Deferred to fast-follow if automated gating is needed. |
| Separate RPAs per build context | One RPA for nightly (auto-release) + one for releases (gated via `block-releases`). Rejected because `block-releases` is per-RPA not per-Snapshot — every build from the Application would create blocked releases on the gated RPA, generating noise without meaningful safety. |
| Glob patterns for tag matching | PaC glob patterns (e.g., `v*`) cannot distinguish `v1.0.0-rc1` from `v1.0.0`. CEL expressions with regex anchoring are required. Verified against PaC source code and hypershift production configuration. |
1 change: 1 addition & 0 deletions hyperfleet/adrs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,3 +88,4 @@ What did we decide? State it plainly.
| [0010](0010-jsonb-provider-agnostic-spec.md) | JSONB for Provider-Agnostic Cluster and NodePool Spec Storage | Active | 2026-04-15 |
| [0011](0011-testcontainers-for-integration-tests.md) | Testcontainers for Integration Tests | Active | 2026-04-16 |
| [0012](0012-hard-delete-mechanism-after-adapter-reconciliation.md) | Hard-Delete Ownership and Execution | Proposed | 2026-04-23 |
| [0013](0013-konflux-build-and-release.md) | Konflux for Container Image Build and Release | Active | 2026-04-21 |
12 changes: 9 additions & 3 deletions hyperfleet/bill-of-artifacts.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,11 +197,17 @@ All core services use testcontainers-go for integration testing and golangci-lin

| Field | Value |
|-------|-------|
| **Reference** | [hyperfleet-release-process.md](https://github.com/openshift-hyperfleet/architecture/blob/main/hyperfleet/docs/hyperfleet-release-process.md) |
| **Release Process** | [hyperfleet-release-process.md](./docs/release/hyperfleet-release-process.md) |
| **Pipeline Design** | [konflux-release-pipeline-design.md](./docs/release/konflux-release-pipeline-design.md) |
| **ADR** | [0013 — Konflux for Container Image Build and Release](./adrs/0013-konflux-build-and-release.md) |

### 6.1 Prow CI
### 6.1 Konflux (Build and Release)

Primary CI system for core services. Presubmit and postsubmit jobs across all core repositories covering unit tests, integration tests, linting, and Helm chart validation. Job definitions are externally managed. Dedicated Prow CI cluster provisioned on GCP.
Container images are built and released via Konflux on the `stone-prd-rh01` cluster. PaC tag-triggered pipelines build images, Tekton Chains signs provenance, and the `rh-push-to-external-registry` pipeline publishes to `quay.io/redhat-services-prod/hyperfleet/`. See [Konflux Release Pipeline Design](./docs/release/konflux-release-pipeline-design.md) for full architecture.

### 6.2 Prow CI

CI system for PR validation and E2E testing. Presubmit and postsubmit jobs across all core repositories covering unit tests, integration tests, linting, and Helm chart validation. Job definitions are externally managed. Dedicated Prow CI cluster provisioned on GCP.

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ This document defines integration test scenarios and acceptance criteria for the
**Related Documentation:**
- [Adapter Framework Design](./adapter-frame-design.md) - Architecture overview
- [Adapter Status Contract](./adapter-status-contract.md) - Status reporting contract
- [Test Release MVP](../../../docs/test-release/test-release-MVP.md) - Testing strategy
- [Test Release MVP](../../../docs/release/test-release/test-release-MVP.md) - Testing strategy

### Testing Strategy

Expand Down
2 changes: 1 addition & 1 deletion hyperfleet/components/api-service/api-versioning.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ The HyperFleet API uses semantic versioning (MAJOR.MINOR.PATCH):
- Changing field types
- Changing endpoint behavior in non-backwards-compatible ways

**Note:** For container image tagging strategy and release management, see [Git and Release Strategy](../../docs/test-release/git-and-release-strategy.md).
**Note:** For container image tagging strategy and release management, see [HyperFleet Release Process](../../docs/release/hyperfleet-release-process.md).

---

Expand Down
14 changes: 14 additions & 0 deletions hyperfleet/docs/release/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Release Documentation

## Start Here

| Document | Purpose |
|----------|---------|
| [Konflux Release Pipeline Design](./konflux-release-pipeline-design.md) | How the build and release pipeline works — architecture, flows, and design decisions |
| [HyperFleet Release Process](./hyperfleet-release-process.md) | The release process — cadence, checklists, branching, bug handling, hotfixes |
| [Glossary](./glossary.md) | Definitions of terms used across the release docs |
| [ADR 0013](../../adrs/0013-konflux-build-and-release.md) | Decision record for adopting Konflux |

## Prow Test and Release

The `test-release/` subdirectory contains Prow-specific docs for CI job setup and E2E testing infrastructure.
66 changes: 66 additions & 0 deletions hyperfleet/docs/release/glossary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
---
Status: Active
Owner: HyperFleet Team
Last Updated: 2026-04-21
---

# Release Glossary

> **Audience:** HyperFleet developers, release owners, and anyone reading the release documentation.

Key terms used across the release docs. Written for someone encountering these systems for the first time.

---

## Konflux and Build Pipeline

| Term | Definition |
|------|------------|
| **Konflux** | Red Hat's build and release platform. Orchestrates container image builds, signing, policy validation, and registry publishing. |
| **PaC** (Pipelines as Code) | Trigger layer. Watches GitHub webhooks and matches events to `.tekton/` pipeline files in component repos via annotations. |
| **Tekton** | Kubernetes-native pipeline framework used by Konflux. Defines Pipelines, Tasks, and PipelineRuns as custom resources. |
| **Tekton Chains** | Controller that automatically signs provenance records (in-toto format) with cosign after every build. Zero configuration needed. |
| **Snapshot** | Immutable record of all component images at specific commits. Created after each build. Releases operate on Snapshots, not individual images. |
| **RPA** (ReleasePlanAdmission) | Defines how releases are processed: component-to-registry mapping, tags, EC policy, release pipeline. Lives in `konflux-release-data`. |
| **ITS** (IntegrationTestScenario) | A test pipeline that runs against a Snapshot before release. Mandatory ITS blocks release on failure. |
| **EC** (Enterprise Contract) | Policy engine validating container images meet compliance requirements. Checks provenance, acceptable bundles, base images. |
| **CEL** (Common Expression Language) | Used in PaC annotations for event matching. RE2 regex via `matches()`. Required for distinguishing RC from release tags. |
| **Provenance** | Signed record proving how and where a container image was built. Enables verification that an image came from a specific commit via Konflux's pipeline. |
| **SBOM** (Software Bill of Materials) | Machine-readable inventory of all packages in a container image. Generated automatically by Konflux builds. |
| **SLSA** (Supply-chain Levels for Software Artifacts) | Framework for build integrity. Konflux achieves Level 3 by default — builds on hardened infrastructure with signed provenance. |

## Prow and Testing

| Term | Definition |
|------|------------|
| **Prow** | Kubernetes-based CI system used by OpenShift. Runs PR validation, E2E tests, and periodic jobs. |
| **ci-operator** | OpenShift CI tool that orchestrates container image builds and test execution. Configured via files in the `openshift/release` repo. |
| **presubmit** | Prow job triggered on PR creation. Runs lint, unit tests, and build checks. Must pass before merge. |
| **postsubmit** | Prow job triggered after code is merged to main. Runs post-merge validations. |
| **periodic** | Prow job triggered on a cron schedule. Used for nightly E2E tests and regression testing. |
| **Gangway** | Prow API that allows external services to trigger Prow jobs programmatically. Used for RC E2E triggering via GitHub Actions. |
| **E2E** (End-to-End) | Tests that validate complete user workflows across all system components deployed to a real environment. |

## Release Process

| Term | Definition |
|------|------------|
| **RC** (Release Candidate) | Pre-release version tagged for validation before GA. Example: `v1.5.0-rc1`. Requires full E2E testing before promotion. |
| **GA** (General Availability) | Final production release version. Example: `v1.5.0`. Signals production-ready status to partner teams. |
| **Feature Freeze** | Point in the release cycle when the release branch is created. No new features after this — only bug fixes. |
| **Code Freeze** | Point after stabilization when only Blocker/Critical fixes are permitted. Strict change control with Release Owner approval. |
| **Release Owner** | Person responsible for coordinating the release: approving cherry-picks, tracking bugs, ensuring release criteria are met. |
| **RELEASE_MANIFEST.yaml** | File in `hyperfleet-release` repo tracking which component versions constitute a validated release. Updated before each RC and GA tag. |
| **Cherry-Pick** | Applying a specific commit from one branch to another. HyperFleet fixes main first, then cherry-picks to the release branch. |
| **Hotfix** | Urgent fix applied to a released version outside the normal release cycle. Patches skip RC — straight to a patch tag. |
| **N-1 Compatibility** | Supporting one version back (e.g., v1.5 compatible with v1.4). |

## Infrastructure

| Term | Definition |
|------|------------|
| **stone-prd-rh01** | Konflux production cluster hosted by Red Hat. Runs the PaC controller, Tekton Chains, and release pipelines for HyperFleet. |
| **Quay** | Red Hat's container registry. HyperFleet images are published to `quay.io/redhat-services-prod/hyperfleet/`. |
| **Pyxis** | Red Hat's container metadata catalog. Images registered here are automatically scanned for CVEs. |
| **ArgoCD** | GitOps-based deployment tool. Partner teams use ArgoCD to consume HyperFleet images from Quay. |
| **konflux-release-data** | GitLab repo containing RPA, constraint, and tenant configuration for Konflux releases. Source of truth for release pipeline config. |
Loading